Illumination-based assistance during extravehicular activity

ABSTRACT

A device is configured to provide assistance during extravehicular activity. The device obtains and operates on first data indicative of one or more objects in a detection space. The device is configured to process the first data for determination of one or more properties for the object(s); evaluate, based on the one or more properties, the object(s) in relation to a set of rules to determine one or more user instructions, and cause an illumination arrangement to provide the one or more user instructions by selective projection of the light in relation to the one or more objects. The device thereby reduces the reliance on cognitive processing by user(s) to take decisions on how to proceed in a situation, and instead the cognitive processing of information about the surroundings in relation to the task at hand is offloaded to the device.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Swedish Patent Application2151351-0, filed Nov. 3, 2021, the content of which is incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to equipment for use inlow-gravity environments and, in particular, to such equipment forproviding assistance to individuals in low-gravity environments.

BACKGROUND ART

Extravehicular activity (EVA) is any activity done by an individualoutside a spacecraft beyond the Earth’s appreciable atmosphere, i.e. ina low-gravity environment. EVAs include spacewalks, surface exploration,maintenance and repair, etc. An individual that travels into space iscommonly known as an astronaut, cosmonaut, or taikonaut.

During EVA, the individual wears a space suit, which is a garmentdesigned to keep a human alive in the harsh environment of outer space.Apart from the overall challenges of moving around in low gravity, theindividual also struggles with limited field of vision, limitedflexibility of the pressurized space suit, and adverse lightningconditions. An individual that performs a task during EVA will face manychallenges. One challenge is to avoid falling when walking on anunexplored surface, since falling may result in undesired contaminationof the space suit by regolith. The fall may also result in damages tothe space suit, and the individual may require assistance to get back onfeet. Another challenge of working in low-gravity environments is totrack and manipulate multiple objects when performing tasks that requirea high level of focus and concentration, for example service and repair.Yet another challenge is to perform any task that requires collaborationamong individuals.

Adverse lighting conditions may be remedied by external lighting or anillumination device attached to the space suit. In this context,US11029160 proposes to attach a projector to the space suit or a vehicleand operate the projector to produce an image on the terrain to provideinformation to an individual. The image may be a grid pattern that helpsthe individual to appreciate the structure of the terrain. The image mayalso be a schematic or the like, which is projected onto an article inthe terrain to help the individual during service or repair of thearticle. The image may also indicate presence of minerals, water or icein the terrain, as detected by a sensor. While being capable ofassisting the individual, the projector system is reliant on theattention and cognitive ability of the individual. Even if astronautsare highly trained and well-prepared, they are in a vulnerable andpotentially stressful situation on EVAs. Further, with the developmentof augmented reality (AR) displays, astronauts are likely to be exposedto increasing amounts of information during EVAs and projectinginformation into the surroundings for cognitive processing by theastronaut may be more confusing than helpful to the astronaut.

BRIEF SUMMARY

It is an objective to at least partly overcome one or more limitationsof the prior art.

Another objective is to provide a technique that provides assistance toone or more individuals during extravehicular activity.

Yet another objective is to provide such a technique that improves theperformance of a task by the one or more individuals.

One or more of these objectives, as well as further objectives that mayappear from the description below, are at least partly achieved by adevice for assisting an individual during extravehicular activity, aspace suit, a computer-implemented method, or a computer-readable mediumaccording to the independent claims, embodiments thereof being definedby the dependent claims.

Still other objectives, as well as features, aspects and technicaleffects will appear from the following detailed description, from theattached claims as well as from the drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a side view of an individual dressed in an example space suit.

FIG. 2 is a block diagram of an example control system for a space suit.

FIGS. 3A-3B are top plan views, partly in section, of example head posesin a helmet of a space suit.

FIG. 4 is a block diagram of an example device for audio-basedassistance of an individual performing an extravehicular activity.

FIG. 5 is a flow chart of an example method for audio-based assistance.

FIG. 6 is a schematic illustration of audio feedback provided by thedevice in FIG. 4 when operated in accordance with the method in FIG. 5 .

FIGS. 7A-7C are flow charts of example procedures in the method of FIG.5 .

FIGS. 8A-8B illustrate example uses of the device for audio-basedassistance.

FIG. 9 is a side view of an individual dressed in a space suitperforming an example task.

FIG. 10 is a block diagram of an example device for providingperformance-related feedback to an individual in relation to a taskperformed by the individual.

FIG. 11 is a flow chart of an example method of providingperformance-related feedback.

FIG. 12 is a block diagram of an example processing system forperforming the method of FIG. 11 .

FIG. 13A is a schematic illustration of an activity sequence definitionused by the processing system in FIG. 12 , and FIGS. 13B-13D are flowcharts of example procedures in the method of FIG. 11 .

FIG. 14A is a graph of example actions determined by the procedure inFIG. 13B, and FIGS. 14B-14C are graphs of time-resolved body pose andgaze direction during a predefined time window in accordance with anexample.

FIG. 15 is a side view of an individual dressed in a space suitcomprising an illumination arrangement.

FIG. 16 is a block diagram of an example device for providingillumination-based assistance.

FIG. 17 is a flow chart of an example method for providingillumination-based assistance.

FIG. 18 is a block diagram of an example processing system forperforming the method of FIG. 17 .

FIG. 19 is a schematic illustration of an example set of rules used inthe method of FIG. 17 .

FIGS. 20A-20G are top plan views of application examples for the set ofrules in FIG. 19 .

FIG. 21 is a block diagram of a machine that may implement methods,procedures and functions as described herein.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Embodiments will now be described more fully hereinafter with referenceto the accompanying schematic drawings, in which some, but not all,embodiments are shown. Indeed, the subject of the present disclosure maybe embodied in many different forms and should not be construed aslimited to the embodiments set forth herein; rather, these embodimentsare provided so that this disclosure may satisfy applicable legalrequirements.

Also, it will be understood that, where possible, any of the advantages,features, functions, devices, and/or operational aspects of any of theembodiments described and/or contemplated herein may be included in anyof the other embodiments described and/or contemplated herein, and/orvice versa. In addition, where possible, any terms expressed in thesingular form herein are meant to also include the plural form and/orvice versa, unless explicitly stated otherwise. As used herein, “atleast one” shall mean “one or more” and these phrases are intended to beinterchangeable. Accordingly, the terms “a” and/or “an” shall mean “atleast one” or “one or more”, even though the phrase “one or more” or “atleast one” is also used herein. As used herein, except where the contextrequires otherwise owing to express language or necessary implication,the word “comprise” or variations such as “comprises” or “comprising” isused in an inclusive sense, that is, to specify the presence of thestated features but not to preclude the presence or addition of furtherfeatures in various embodiments.

As used herein, the terms “multiple”, “plural” and “plurality” areintended to imply provision of two or more elements, whereas the term a“set” of elements is intended to imply a provision of one or moreelements. The term “and/or” includes any and all combinations of one ormore of the associated listed elements.

It will furthermore be understood that, although the terms first,second, etc. may be used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first elementcould be termed a second element, and, similarly, a second element couldbe termed a first element, without departing the scope of the presentdisclosure.

Well-known functions or constructions may not be described in detail forbrevity and/or clarity. Unless otherwise defined, all terms (includingtechnical and scientific terms) used herein have the same meaning ascommonly understood by one of ordinary skill in the art to which thisdisclosure belongs.

Like numerals refer to like elements throughout.

Before describing embodiments in more detail, a few definitions will begiven.

As used herein, “extravehicular activity”, EVA, refers to any activitydone by an individual outside a spacecraft beyond the Earth’sappreciable atmosphere.

As used herein, “audible sound” refers to a sound that falls within thehearing range of humans, typically 20-20,000 Hz.

As used herein, “visible light” refers to electromagnetic radiation thatis visible to the human eye, typically in the wavelength range of300-1100 nm.

Embodiments relate to various aspects of devices and methods forassisting individuals during EVA. The following description is separatedinto Chapters 1-3 dealing with inventive concepts for EVA assistance.The separation into chapters is made for clarity of presentation anddoes not imply that the concepts cannot be combined. On the contrary,the concepts and embodiments thereof may indeed be combined in variousconstellations to achieve corresponding synergistic effects, as will beappreciated by the persons skilled in the art.

For context only, FIG. 1 schematically shows an individual 1 wearing aspace suit 2 for EVA. Space suits for EVA are designed to provide stableinternal pressure and enable the individual to move its limbs, and toshield against different types of radiation as well as micrometeoroids.Various equipment may be attached to or integrated with the space suit2, for example a portable life support system (PLLS) 3 and a maneuveringunit 4. The PLLS 3 may be configured to, for example, regulate suitpressure, provide breathable oxygen, remove carbon dioxide, humidity,odors, and contaminants from breathing oxygen, cool and recirculate gasor liquid through the space suit. The maneuvering unit 4 may beconfigured to provide propulsion power for controlled movement of theindividual in free space. In the illustrated example, the space suit 2comprises a garment portion 9 and a helmet 10. The helmet 10 isreleasably attached to the top of the garment portion 9 by a fastenerarrangement 8. The helmet 10 is typically rigid and comprises atransparent visor 12 that defines the available field of vision of theindividual.

The space suit 2 and any auxiliary equipment 3, 4 may be seen to be partof a space suit system, SSS, which is electronically controlled. The SSSincludes a control device that comprises logic configured to control theoperation of the SSS. The logic may be implemented in hardware, softwareor a combination of both hardware and software.

A detailed example of an SSS is depicted in FIG. 2 . In the illustratedexample, the SSS includes one or more processors 102, a memorycontroller 103, a peripherals interface (I/F) 104, a memory 105, asource 106 of electrical power, an arrangement 107 of internal sensors,an arrangement 108 of external sensors, audio control circuitry 109,visual control circuitry 110, a communication interface (I/F) 111, oneor more manual controls 112 for user input, a microphone 113, a speakersystem 114, a display system, optionally configured for augmentedreality (AR), light control circuitry 116, one or more light sources117, thrust control circuitry 118, a propulsion system 118′ and fluidcontrol circuitry 119. One or more of these components may communicateover one or more communication buses or signal lines as represented byarrows in FIG. 2 .

FIG. 2 is only one example of an SSS system. An SSS may thus includemore or fewer components than those shown in FIG. 2 , may combine two ormore components as functional units, or a may have a differentconfiguration or arrangement of the components. Some of the variouscomponents shown in FIG. 2 may be implemented in hardware, software or acombination of both hardware and software, including one or more signalprocessing and/or application-specific integrated circuits.

In some embodiments, the processor 102 runs or executes various softwareprograms and/or sets of instructions stored in memory 105 to performvarious functions of the SSS and to process data. In some embodiments,processor 102 includes one or more of a CPU (“Central Processing Unit”),a DSP (“Digital Signal Processor”), a microprocessor, a microcontroller,an ASIC (“Application-Specific Integrated Circuit”), a combination ofdiscrete analog and/or digital components, or some other programmablelogical device, such as an FPGA (“Field Programmable Gate Array”). Thememory 105 may include one or more computer-readable storage mediums,such as high-speed random access memory, and/or non-volatile memory,such as one or more magnetic disk storage devices, flash memory devices,or other non-volatile solid-state memory devices. Access to memory 105by other components such as the processor(s) 102 and the peripherals I/F104 may be controlled by the memory controller 103. The peripherals I/F104 may couple input and output peripherals to the processor(s) 102 andmemory 105. In some embodiments, the processor(s) 102, the memorycontroller 103 and the memory 105 are part of the above-mentionedcontrol device 101, which is indicated by dashed lines and may or maynot be integrated on a single chip.

In some embodiments, software components stored in memory 105 include anoperating system, and a set of software modules or applications. Thesoftware modules may correspond to a set of instructions for performingone or more functions by use of components shown in FIG. 2 . Forclarity, such software modules are not shown in FIG. 2 . The operatingsystem may comprise various software components and/or drivers forcontrolling and managing general system tasks (for example, memorymanagement, storage device control, power management, etc.) andfacilitates communication between various hardware and softwarecomponents.

In some embodiments, the SSS includes a source 106 of electrical power(“power system”) for powering its various components. The power system106 may include a power management system, one or more local powersources such as battery, fuel cell, photovoltaic cells, etc., arecharging system, a power failure detection circuit, a power converteror inverter, a power status indicator and any other componentsassociated with the generation, management and distribution of power ina computerized device.

In some embodiments, the SSS includes an arrangement 107 of internalsensors, which are configured to detect and/or measure properties of theindividual and/or the interior of the space suit. The arrangement 107may, for example, comprise sensors for measuring gas pressure, gascomposition, temperature, humidity, heart activity, blood pressure,stress level, etc. In some embodiments, the arrangement 107 comprises atleast one of a head tracking device or a gaze tracking device. The headtracking device is configured to determine the momentary pose of thehead of the individual inside the helmet and may be based on anyconventional technique such as computer vision and/or inertial sensorsdirectly or indirectly attached to the head. In another alternative, thepose of the head is determined by the use of cooperating elements on thehead and the inside of the helmet, for example magnets and magneticfield sensors, or light emitters and light sensors. The gaze trackingdevice is configured to determine the momentary gaze direction of theindividual, by measuring eye positions and eye movement. The gazetracking device may be based on any conventional technique such asoptical tracking by computer vision, eye-attached tracking or electricalpotential measurement by electrodes placed around the eyes. In someembodiments, the arrangement 107 comprises sensors for measuring themomentary body pose of the individual (“body pose sensors”). The bodypose may be given in any detail and may thus define the relativeposition of any number of limbs. The body pose sensors may comprisewearable inertial sensors directly or indirectly attached to variouslimbs of the individual and/or to various portions of the space suit.Alternatively or additionally, the body pose sensors may comprise strainsensors directly or indirectly attached to the individual and/or thespace suit.

In some embodiments, the SSS includes an arrangement 108 of externalsensors, which are configured to detect and/or measure properties in thesurroundings of the individual. The arrangement 108 may, for example,comprise sensors for measuring environmental parameters such as gascomposition, temperature, etc. In some embodiments, the arrangement 108comprises one or more sensors for detection of objects and/or one ormore properties of objects in a detection space around the individual.The detection space may have any shape and extent. In some embodiments,the detection space is omnidirectional. Such an arrangement 108 may beconfigured to generate measurement data representative of one or more ofa position, a shape, a distance, a temperature, or a composition of anobject. Examples of sensors in the arrangement 108 include visionsensors that provide a two-dimensional (2D) or three-dimensional (3D)representation within a field of view. The vision sensor may be an arraydetector or camera for radiation detection in one or more wavelengthregions, for example a visual camera, a thermal camera, a multispectralcamera, etc. Other examples of vision sensors include Light Detectionand Ranging (LIDAR) systems, Radio Detection and Ranging (RADAR)systems, ultrasonic sensors, etc.

In some embodiments, the SSS comprises audio control circuitry 109,which is configured to allow the individual to communicate with otherindividuals and/or generate voice commands for a voice-controlleddevice. The other individuals may be located inside or outside of aspacecraft or in a mission control center, for example on Earth. Theaudio control circuitry 109 is connected to one or more microphones 113and a speaker system 114 inside the space suit. The microphone(s) 113and the speaker system 114 may be attached to the head of the individualand/or to the inside of the helmet.

In some embodiments, the SSS comprises visual control circuitry 110,which is configured to visually present information to the individual.The visual information may represent data generated by the arrangements107, 108 or data received via the communication I/F 111. In someembodiments, the visual control circuitry 110 is connected to a displaydevice 115, which may be integrated into or associated with the visor(cf. 12 in FIG. 1 ) and/or be worn on the individual’s head or eyes. Thedisplay device 115 may or may not be configured to provide augmentedreality (AR). For example, the display device 115 may comprise one ormore of a head-up-display (HUD), bionic contact lenses, a virtualretinal display, AR enabled eyeglasses, etc. In some embodiments, thevisual control circuitry 110 is connected to light control circuitry116, which is operable to control one or more light sources 117 toproject visible light into the surroundings of the space suit. The lightcontrol circuitry 116 may control the spatial distribution and/or thewavelength of the light provided by the light source 117. In oneexample, the light source 117 is operable to selectively direct a lightbeam into the surroundings, for example by use of a conventional beamscanner. In another example, the light source 117 is operable togenerate and project a static beam into the surroundings.

In some embodiments, the SSS comprises a communication I/F 111, which isconfigured for reception of communication signals, for example, in theform of electromagnetic signals. The communication I/F 111 may enablelocal communication with a spacecraft or other space suits, as well asremote communication with a mission control center. The transmission andreception of communication signals may be carried out wirelessly, forexample via a radio frequency (RF) transceiver. In some embodiments, thecommunication I/F 111 includes RF circuitry, which is configured toconvert electrical signals to/from electromagnetic signals andcommunicate with communication networks and other communication devicesvia the electromagnetic signals. The RF circuitry may include well-knowncircuitry for transmission and reception of communication signals,including but not limited to an antenna system, an RF transceiver, oneor more amplifiers, a tuner, one or more oscillators, a digital signalprocessor, a CODEC chipset, etc. The wireless communication may use anyavailable communication standards, protocols and technologies.

In some embodiments, the SSS comprises one or more manual controls 112for user input. The manual controls may comprise one or more physicalbuttons (e.g., push buttons, rocker buttons, etc.), dials, levers, keys,slider switches, joysticks, click wheels, and so forth.

In some embodiments, the SSS comprises thrust control circuitry 118,which is configured to control the thrust generated by a propulsionsystem 118′, which may be included in the SSS (cf. maneuvering unit 4 inFIG. 1 ). The thrust control circuitry 118 may be operable to controlthe magnitude and direction of the thrust. The propulsion system 118′may be operable to release a fluid to generate the desired thrust. In anon-limiting example, the fluid comprises one or more of oxygen ornitrogen. The propulsion system 118′ may be activated to move theindividual in space, for example during a spacewalk.

In some embodiments, the SSS comprises fluid control circuitry 119,which is configured to control the supply and/or removal of one or morefluids within the space suit. The fluid control circuitry 119 maycorrespond to the PLLS 3 in FIG. 1 .

1. Audio-Based Assistance During Extravehicular Activity

This part of the disclosure relates to a technique of providingassistance to an individual or user during EAV to improve the user’sperception of the surrounding environment. This part of the disclosureis particularly, but not exclusively, related to an inventive concept ofindicating presence of an object in the surrounding environment bygenerating an audible sound with a perceived origin, for the user, thatcorresponds to the location of the object in relation to the user’shead. The audio-based assistance is thereby intuitive to the user andmay be provided without occluding or obstructing the user’s field ofvision.

The inventive concept will be further explained with reference to FIGS.3A-3B, which are top plan views onto a head 20 inside an example helmet10 of a space suit. For illustration purposes, the helmet 10 is shown incross-section. The direction of the head 20 is indicated by the locationof the nose 21 and the ears 22. As shown, the helmet 10 defines aprotective shell 11 around the head 20. The shell 11 is spaced from thehead 20, which is thereby movable in relation to the helmet 10. This isa common property of space suit helmets. Conventionally, the helmet 10is shaped as a spheroidal dome in order to balance the need for field ofview, pressure compensation, and low weight.

In the example of FIGS. 3A-3B, an external monitoring sensor 31 isarranged on the helmet 10. The term external does not imply that themonitoring sensor 31 needs to be located on the outside of the spacesuit but rather that it is configured to generate external sensor data,ESD, which is indicative of a detection space outside and around thehelmet 10. In some embodiments, the monitoring sensor 31 is configuredto generate measurement data indicative of objects in the detectionspace. Such measurement data is at least indicative of the position ofthe object in relation to the monitoring sensor 31 and may also beindicative of a property of the object, such as its shape, itstemperature, or its composition. The monitoring sensor 31 may be part ofthe arrangement 108 as depicted in FIG. 2 and may be implemented as avision sensor.

The monitoring sensor 31 need not be arranged on the helmet 10, but maybe arranged anywhere on the space suit. Further, a plurality ofmonitoring sensors 31 may be arranged to provide the ESD. In oneexample, at least some of the monitoring sensors 31 are identical andarranged at different locations on the space suit, for example toincrease the detection space. In another example, at least some of themonitoring sensors 31 are of different types, for example to increasethe diversity of the ESD and increase the ability to detect objectsand/or properties of objects.

In the example of FIGS. 3A-3B, a head tracking device 32 is arranged onthe helmet 10 to generate head pose data, HPD, which is indicative ofthe momentary head pose. As used herein, “head pose” refers to theorientation, and optionally also the location, of the head 20 inside andin relation to the helmet 10. In the illustrated example, the headtracking device 32 is arranged inside the helmet 10 but it mayalternatively be located on the outside. The head tracking device 32 maybe part of the arrangement 107 as depicted in FIG. 2 .

FIGS. 3A-3B also illustrate the provision of an array of speakers 33that are distributed on the inside of the helmet 10. The number ofspeakers 33 in the array is at least two to enable spatialization of thegenerated sound (below). In a variant, the speakers 33 are arranged in aheadset located on the ears 22 of the user. The array of speakers 33 maybe part of the speaker system 114 as depicted in FIG. 2 .

To facilitate the following discussion, first and second coordinatesystems C1, C2 are indicated in FIGS. 3A-3B. The first coordinate systemC1 is fixed in relation to the sensor 31. The second coordinate systemC2 is fixed in relation to the head. The coordinate systems C1, C2 areshown as Cartesian systems of (x,y,z) axes and (x′,y′,z′) axes,respectively, but any other coordinate system may be used, for examplepolar coordinate systems.

The helmet 10 also comprises a transparent visor 12, which allows theuser to view the surroundings in front of the helmet 10. In FIGS. 3A-3B,the dashed lines represent the limits of the central vision of the user,when keeping the head in a fixed position and only moving the eyes.Thus, in FIGS. 3A-3B, IFV designates the available span of centralvision. The central vision is also known as foveal vision and allows theuser to distinguish detail, color, and shape of objects. Outside thecentral vision, in the regions of paracentral vision, mid-peripheralvision and far-peripheral vision (collectively denoted “peripheralvision” herein), humans find it difficult to distinguish betweenobjects. Humans therefore tend to move their head to scan thesurroundings. Further, upon perceiving that an object is present withinthe peripheral vision, humans tend to turn their head towards the objectto bring it into central vision.

In FIG. 3A, the user faces the visor 12 and has an IFV that isunobstructed by the helmet 10, at least in the horizontal direction. InFIG. 3B, the individual has turned the head 20 to the left, and the IFVis partially delimited by the non-transparent helmet material to theleft of the visor 12. When also considering the peripheral vision of theuser, it is realized that the helmet 10 will obstruct at least part ofthe peripheral vision for all orientations of the head 10 within thehelmet 10. Thus, the field of vision including peripheral vision may beinvariant of head pose.

To visually scan a larger portion of the surroundings, the user willhave to twist the torso or the entire body to turn the helmet 10. Thismay be quite difficult and takes time in low gravity. Given the visualconstraints of the helmet 10, the user is vulnerable to emerging objectsthat may pose a hazard to the user. Also, the visual constraints maymake it quite cumbersome for the user to perform various tasks inrelation to one or more surrounding objects. It should be borne in mindthat objects, and also the user, gravitate towards heavier objects overtime in free space as a result of the low gravity. Thus, the combinationof low gravity and visual constraints may cause an undesirable change inthe location of an object to go unnoticed by the user. There is thus ageneral desire to improve the awareness of the surroundings for a userthat wears a space suit in a low gravity environment.

This objective is at least partly achieved by a device 120 shown in FIG.4 . The device 120 is part of a system 100, which also includes themonitoring sensor(s) 31, the head tracking device 32 and the array ofspeakers 33. The device 120 comprises processor circuitry 101, which maycorrespond to the control device 101 in FIG. 2 , a first input device120A configured to receive the ESD from the monitoring sensor(s) 31, anda second input device 120B configured to receive the HPD from the headtracking device 32. The input devices 120A, 120B (“inputs”) may compriseany conventional hardware interface, optionally combined with software,for receiving input signals. The device 120 further comprises an outputdevice 120C (“output”) configured to provide audio control data, ACD, tothe array of speakers 33, optionally via the audio control circuitry 109in FIG. 2 . In FIG. 4 , the monitoring sensor 31 is part of the sensorarrangement 108, the head tracking device 32 is part of the sensorarrangement 107, and the speakers 33 are part of the speaker system 114.

The device 120 in FIG. 4 may be included in the space suit 2, forexample by manufacture or retrofitting. The device 120 may be a unitarycomponent or an assembly of separate parts. It is also conceivable thatthe device 120 includes one or more additional components of the system100, for example the array of speakers 33, the monitoring sensor(s) 31,or the head tracking device 32. In some embodiments, the helmet 10 isalso part of the device 120.

FIG. 5 is a flowchart of an example method 140 for assisting a userduring EVA. The method 140 is computer-implemented and may be performedby the device 120 in FIG. 4 . The method 140 may be implemented byhardware or a combination of hardware and software, for example byprogram instructions stored in a memory in the device 120. The method140 comprises steps 141-145, which may be repeated over time to providecontinuous assistance to the user.

In step 141, first data and second data are obtained. The first data isindicative of the detection space around the user. The first datacorresponds to the ESD in FIG. 4 and is obtained by the device 120 viainput 120A. The detection space is defined by the volume in which theone or more monitoring sensors 31 are responsive to objects. The seconddata is indicative of the user’s head pose relative to the helmet 10.The second data corresponds to the HPD in FIG. 4 and is obtained by thedevice 120 via input 120B.

In step 142, the first data is processed for detection of an object ofrelevance to the user in the detection space. The object of relevancemay be any type of object in a broad sense. The object may thus be atactile object, such as a man-made implement, a piece of terrain,another user in a space suit, space debris, a meteoroid, etc. The objectmay also be an impalpable object, such as a region ofincreased/decreased temperature, a formation of gas, etc. Step 142 maydetect any number of objects of relevance. In some embodiments, any andall objects that are detected within the detection space may beconsidered to be an object of relevance in step 142. In otherembodiments, the object of relevance is confined to one or more specificcategories of objects (below).

In step 143, a first position of the respective object with respect tothe helmet 10 is determined based on the first data. In the example ofFIGS. 3A-3B, the first position is determined in the first coordinatesystem C1, and may be given by (x,y,z) coordinates. The first positionmay represent any detectable feature of the object, for example acorner, a geometric center, etc. It is also to be understood that step143 may determine more than one first position for an object. Althoughthe first position is typically a 3D position given in three dimensions,it may be a 2D position in some embodiments. As used herein, a positionthat is determined “with respect to the helmet” implies that theposition is determined in relation to a reference point on the spacesuit. Thus, the reference point need not, but may, be located on thehelmet.

In practice, steps 142 and 143 may be merged so that the first positionof the object is determined as part of the processing for objectdetection.

In step 144, a second position of the object with respect to the user’shead 20 is determined based on the first position from step 143 and thesecond data (HPD) from step 141. As used herein, a position that isdetermined “with respect to the head” implies that the position isdetermined in relation to a reference point on the head 20. By step 144,the second position is determined to locate the object in relation tothe head 20 rather than the space suit 2 or the helmet 10. The skilledperson understands that the determination in step 144 is feasible sincethe second data defines a spatial relation between the head 20 and thehelmet 10, and the first position defines a spatial relation between theobject and the helmet 10. If more than one first position is determinedfor the object in step 143, a number of second positions may bedetermined in step 144, for all or a subset of the first positions. Inthe example of FIGS. 3A-3B, the second position is determined in thesecond coordinate system C2, and may be given by (x′,y′,z′) coordinates.

In step 145, the array of speakers 33 is caused to generate an audiblesound with a spatial origin given by the second position. Thereby, theuser will be informed not only about the presence of the object but alsoof its location in relation to the user’s head. In the context of FIG. 4, step 145 results in the audio control data, ACD, which is output bythe control device 120. In some embodiments, the spatial origin is setto coincide with the second position. In other embodiments, the spatialorigin is set in the direction of the object from the user’s head 20,for example somewhere along a line connecting the head 20 and the secondposition. For the latter embodiments, the audible sound will notreproduce the distance to the object but still inform the user about thedirection to the object.

The generation of audible sound with a specific spatial origin iscommonly known as “3D audio spatialization” and involves manipulatingthe audio signals to different speakers in the array of speakers 33, soas to achieve a desired virtual placement of the spatial origin of theresulting audible sound.

In a non-limiting example, 3D audio spatialization is achieved by use ofso-called head-related transfer functions (HRTFs) for speakers with aknown placement. An HRTF module may be configured to obtain HRTFs fortwo or more speakers 33, based on the second position. The HRTF modulemay retrieve the HRTFs from an HRTF database, which associates HRTFswith positions. Such a database may be generic for all users or tailoredto a specific user. The HRTF module may be configured to select the bestmatch of HRTFs from the database given the second position, or performan interpolation among the HRTFs in the database based on the secondposition. The selected HRTFs are then operated on a base signal togenerate audio signals for the individual speakers. This operation maybe done in the time domain or the frequency domain, as is well known inthe art.

In some embodiments, the 3D audio spatialization is implemented by thecontrol device 120, and the ACD is provided in the form of audio signalsfor individual speakers 33. In other embodiments, the 3D audiospatialization is implemented by the audio control circuitry 109 in FIG.2 , and the ACD is merely indicative of the spatial origin. Partitioningof the 3D audio spatialization between the device 120 and the audiocontrol circuitry 109 is also conceivable.

The method in FIG. 5 will be further illustrated in FIG. 6 , whichcorresponds to FIG. 3A and shows an object 40 that is located within thedetection space of the monitoring sensor 31. The object 40 is detectedin step 142, and a first position of the object 40 is determined inrelation to the first coordinate system C1 in step 143. In theillustrated example, the first position designates a corner 41 on theobject 40. The head pose in relation to the helmet 10, as thus thespatial relation between the first and second coordinate systems C1, C2,is given by second data provided by the head tracking device 32. In step144, the first position in the first coordinate system C1 is convertedinto the second position in the second coordinate system C2 based on thesecond data. For example, a conventional 3D transformation matrix may bedefined based on the second data and operated on the first position. Forexample, as indicated in FIG. 6 , the second position may be given by adistance r′, an azimuth angle ϕ′ in the (x′,y′) plane and an elevationangle (not shown) in the (y′,z′) plane. In step 145, as indicated bypressure wave icons in FIG. 6 , the speakers 33 are activated togenerate audible sound with an origin at the second position in thesecond coordinate system C2.

In some embodiments, the time response of the measured head pose is setto provide a desired user experience. If the time response is high, thelocation of the audible sound may instantly represent the head pose. Ifthe time response is low, the impact of fast or temporary head movementsmay be suppressed. The time response of the measured head pose may beset by a low-pass filter in the sensor arrangement 107 and/or in thedevice 120.

FIG. 7A is a flowchart of an example procedure that may be part of step142 in the method 140. As will be explained further below, results fromstep 142 may be used by step 145 when generating the audible sound, forexample to diversify the feedback provided by audible sound. In step142A, the first data is processed for object detection. As noted, thefirst data may originate from plural monitoring sensors 31, and the datafrom different monitoring sensors 31 may be merged or fused as part ofstep 142, for example to determine multiple properties of an object or a3D shape of the object. In some embodiments, the first data comprises 2Drepresentations (“digital images”) and step 142 involves processing thedigital images by one or more conventional algorithms for objectdetection, which may or may not involve object classification (cf. step142B). Non-limiting examples of such algorithms include various machinelearning-based approaches or deep learning-based approaches, such asViola-Jones object detection framework, SIFT, HOG (Histogram of OrientedGradients), Region Proposals (RCNN, Fast-RCNN, Faster-RCNN, GCNN), SSD(Single Shot MultiBox Detector), You Only Look Once (YOLO, YOLO9000,YOLOv3), RefineDet (Single-Shot Refinement Neural Network for ObjectDetection), and RetinaNet. Step 142 may also involve conventionalfeature detection algorithm(s), for example image processing techniquesthat are operable to detect one or more of edges, corners, blobs,ridges, etc. in digital images. Non-limiting examples of featuredetection algorithms comprise SIFT (Scale-Invariant Feature Transform),SURF (Speeded Up Robust Feature), FAST (Features from AcceleratedSegment Test), SUSAN (Smallest Univalue Segment Assimilating Nucleus),Harris affine region detector, and ORB (Oriented FAST and RotatedBRIEF).

Step 142B performs object classification, based on the first data and/orthe objects detected in step 142A. Step 142B may thus provide at leastone object category for a detected object of relevance. The objects maybe classified into different object categories based on any detectableproperty given by the available monitoring sensors 31, for exampleshape, temperature, composition, speed, etc. The objects may also beclassified based on the above-mentioned object features, if detected instep 142A. As noted, in some embodiments, step 142B may be performed asan integral part of an algorithm used in step 142A.

Step 142C processes at least one of the first data, the objects detectedin step 142A, or the object categories determined in step 142B forhazard detection. Step 142C thereby evaluates if the object poses ahazard or risk to the user. Step 142C may thus indicate whether anobject is associated with a hazard or not. The presence of hazard maydepend on context. For example, during ground exploration in space,unstable ground, steep hills, cliffs, large rocks, sharp objects, spaceexploration vehicles, buildings, etc. may pose a risk to the user.During a spacewalk, meteorites, space debris, sharp objects, firingthrusters, etc. may pose a risk to the user.

Step 142D processes at least one of the first data, the objects detectedin step 142A, or the object categories determined in step 142B forhazard classification. Hazards may be classified into different hazardcategories based on any detectable property given by the availablemonitoring sensors 31, for example shape, size, temperature,composition, speed, movement direction, acceleration, etc. Step 142D maythus provide at least one hazard category for a detected object ofrelevance.

In FIG. 7A, steps 142B-142D are optional and may be implemented independence on the functionality to be provided by step 145. It may alsobe noted that any one of steps 142A-142D may be performed, at leastpartly, by a respective monitoring sensor 31, by a pre-processing unit(not shown) in the sensor arrangement 108 in FIG. 4 , or by cooperativeprocessing among plural monitoring sensors 31. Thus, with reference toFIG. 4 , the first data (ESD) may include raw sensor data from themonitoring sensor(s) 31, pre-processed data, or an indication of objectdetection, optionally together with an indication of object categoryand/or detected hazard and/or hazard category. In the context of FIG. 4, when step 142 is performed by the device 120, the processing of theESD by the device 120 may range from a full processing of raw sensordata in accordance with one or more of steps 142A-142D as describedhereinabove, to an extraction of one or more indications of detectedobject, object category, detected hazard or hazard category from theESD. A similar partitioning of processing is possible for the positiondetermination in step 143. In the context of FIG. 4 , when step 143 isperformed by the device 120, the processing of the ESD by the device 120may range from a full processing of raw sensor data to extracting anindication of the position of the detected object from the ESD.

FIG. 7B is a flowchart of an example procedure that may be part of step145 in the method 140. The procedure in FIG. 7B comprises steps 151-155,which are all optional and may be combined in any way. Common to thesteps 151-154 is that the audible sound is generated with adiscriminable characteristic that is indicative of the object. Thecharacteristic is “discriminable” in the sense that it is audible anddistinguishable from other sounds that may be generated inside thehelmet by the speaker system (cf. 114 in FIG. 2 ). To the extent thatthe purpose of steps 151-154 is to convey information in addition to theposition of the object, the discriminable characteristic is an audibleproperty of the audible sound other than the spatial origin. Forexample, the audible property may be a frequency or combination offrequencies, a duration, a pattern of sound being on and off, aloudness, a timbre, etc., or any combination thereof.

In step 151, the discriminable characteristic is set to indicate aproperty of the detected object. In some embodiments, the discriminablecharacteristic is indicative of the object category, as determined bystep 142B, and differs between different object categories. This allowsthe user to perceive both location and type of object from the audiblesound. In some embodiments, the discriminable characteristic isindicative of whether the detected object is associated with a hazard ornot, as determined by step 142C. This allows the user to perceive thatthere is a potential risk in the detection space and the location of therisk. In some embodiments, the discriminable characteristic isindicative of the hazard category, as determined by step 142D, anddiffers between different hazard categories. This allows the user toperceive that there is a potential risk in the detection space, as wellas the degree and location of the risk.

In step 152, the discriminable characteristic is modified as a functionof a trajectory of the object, assuming that the method 140 comprises astep 152A of determining the trajectory of the detected object based onthe first data, ESD. Step 152A need not be part of step 152 but may beperformed at any time after step 141, when the first data has beenobtained. The trajectory is a time sequence of positions of the object,for example in the first coordinate system C1. The trajectory may be anactual trajectory followed by the object up until a current time pointand/or an estimated future trajectory of the object. A future trajectorymay be estimated in well-known manner, for example based on firstpositions, speed or acceleration. FIG. 8A illustrates a user who isfacing away from an object 40, which is detected by the sensor 31 andfound to be on a future trajectory T1. In some embodiments, step 152 maymodify the discriminable characteristic to indicate whether an actual orfuture trajectory is directed towards or away from the user, or toindicate a likelihood that a future trajectory intersects a futuretrajectory of another detected object or intersects with the user. Inanother example, the discriminable characteristic is set to differbetween different types of trajectories.

In step 153, the discriminable characteristic is modified as a functionof a distance between the detected object and the user and/or a movementdirection of the detected object in relation to the user. For example,the discriminable characteristic may be modified to increasingly capturethe user’s attention as the distance decreases. Similarly, thediscriminable characteristic may be modified to increasingly capture theuser’s attention as the movement direction targets the body of the user.

In step 154, the discriminable characteristic is modified as a functionof head pose, for example the orientation of the head relative to thehelmet as given by the second data, HPD. The head pose is indicative ofthe user’s focus. Generally, step 154 allows the discriminablecharacteristic to be adjusted based on the likelihood that the user isaware of the detected object. Thus, step 154 may be performed to directthe user’s attention to objects that are unlikely to be visuallydetected by the user. Such objects may be hidden from view, by thehelmet or by one or more other objects. Reverting to FIG. 8A, upondetecting that the user’s head 20 is facing away from the object 40, andthe array of speakers (not shown) in the helmet 10 may be caused togenerate the audible sound with a discriminable characteristic that isspecific to hidden objects or to objects located behind the user.

In some embodiments, the discriminable characteristic is modified independence of a relation between the detected object and the user’sfield of vision. The field of vision may be static or dynamic. A dynamicfield of vision changes with head pose of the user, and a static fieldof vision is fixed and independent of head pose. As mentioned above, thefield of vision may differ with head pose, for example if the field ofvision is the available span of central vision through the visor 12 foreach head pose, as indicated by IFV in FIGS. 3A-3B. As also mentionedabove, the field of vision may be independent of head pose when alsotaking the user’s peripheral vision into account.

In some embodiments, as exemplified in FIG. 7B, the method 140 maycomprise a step 154A of calculating the user’s field of vision throughthe visor 12 based on the second data, HPD, and a step 154B ofdetermining the relation between the detected object 40 and the field ofvision. Step 154A thereby determines a dynamic field of vision. Steps154A-154B need not be part of step 154 but may be performed at any timeafter step 141, when the second data has been obtained. It is realizedthat steps 154A-154B may increase the diversity and/or improve therelevance of the audible feedback to the user.

In some embodiments, not illustrated in FIG. 7B, the field of vision isstatic and defined in relation to the helmet 10. Thereby, the relationbetween the detected object and the field of vision may be determined byevaluating the first position of the object in relation to a fixeddefinition of the field of view in the first coordinate system C1 (FIG.6 ). Such embodiments may also increase the diversity and/or improve therelevance of the audible feedback to the user.

In some embodiments, which are applicable to both static and dynamicfields of vision, the audible sound is generated with differentdiscriminable characteristics if the object is within or outside thefield of vision. For example, the audible sound may be softer orentirely turned off if the object is within the field of vision. If theobject is outside the field of vision, the audible sound may begenerated to be more noticeable.

In step 155, the array of speakers 33 is caused to verbalize object datarepresentative of the detected object. As used herein, “verbalize”infers that a message of one or more words is spoken. The use ofverbalization may further improve the user’s perception of objects orrisks in the detection space. It may be noted that different subsets ofthe array of speakers may be activated for the verbalization and thegeneration of the spatialized sound. In some embodiments, the objectdata comprises a characterization of the detected object, for example anobject class, a hazard class, a temperature, a composition, etc. In someembodiments, the object data comprises a distance to the detected objectand/or a direction to the detected object. In some embodiments, theobject data comprises a movement instruction to the user, for example acommand to stop or move away.

It may be noted that at least some steps in the procedure of FIG. 7B maybe implemented to warn the user about objects posing a risk to anotheruser, who is located within the detection space of the user and therebydetected in step 142 (FIG. 5 ). The user may use regular audiocommunication, light signaling, etc., to warn the other user about therisk. For example, hazard detection and/or hazard classification insteps 142C, 142D (FIG. 7A) may be made for the other user as well.Thereby, in step 151, the audible sound may be generated with adiscriminable characteristic indicating to the user that the other useris at risk, and optionally the type of risk. In another example, in step152, the discriminable characteristic may be modified as a function ofthe trajectory in relation to the other user. In yet another example, instep 153, the discriminable characteristic may be modified as a functionof the distance and/or movement direction in relation to the other user.In all examples, the audible sound may be generated with a spatialorigin at the other user or the hazardous object, or both.

FIG. 7C is a flowchart of an example procedure 146 that may be includedin the method 140, for example after step 145. The example procedure 146is provided to guide the user along a path, for example in case of anemergency or during ground exploration. An example of an impendingemergency is shown in FIG. 8B. Here, a user 1 performs a task in freespace outside a spacecraft 50. The space suit comprises the device 120(FIG. 4 ), which is operated to detect risks in the detection spacearound the user 1 and signal such risks by spatialized audible sound,and optionally by verbalization. In the illustrated example, the device120 detects a swarm of meteoroids 40 approaching the user 1. Upondetection of that the swarm poses a risk to the user, the device 120 maydetermine a path P1 for the individual 1 away from the area at risk ofbeing hit by the swarm. The path P1 may be predefined or determineddynamically, for example based on the first data. Generally, the path P1extends from a first location to a second location. In the illustratedexample, the path P1 leads along a ladder 51 to a hatch 52 for entryinto the spacecraft 50. Turning to FIG. 7C, the procedure 146 comprisessteps 161-163. In step 161, a waypoint on the path P1 is determinedbased on the first data (ESD) obtained in step 141. The waypoint, whichis represented as W1 in FIG. 8 , is determined with respect to thehelmet 10, by analogy with step 142. In step 162, a position of thewaypoint W1 with respect to the head of the individual is determinedbased on the second data (HPD) obtained in step 141, by analogy withsteps 143-144. In step 163, the array of speakers in the helmet 10 iscaused to generate an audible sound with a spatial origin at or in thedirection of the waypoint W1 with respect to the head, by analogy withstep 145. Thereby, the user is guided by the audible sound along thepath P1. It is also conceivable that the array of speakers is caused toverbalize the path P1 in step 163.

One advantage of some embodiments described in the foregoing is thatobstacles and hazards outside the user’s field of vision are presentedthrough audio, allowing the user to simultaneously process visualinformation within the user’s field of vision. In comparison, an ARsystem in unable to indicate the position of objects outside the user’sfield of vision. Thus, embodiments described herein may replace orsupplement an AR system.

Another advantage of some embodiments is that the user is guided tosafety if vision is lost, for example if the user is blinded or thehelmet visor is compromised.

Further, some embodiments enable the user to be informed aboutcharacteristics that are imperceptible to the human eye. For example,some embodiments may help the user to distinguish between hot and coldobjects.

In the following, clauses are recited to summarize some aspects andembodiments as disclosed in the foregoing.

C1. A device for assisting an individual during extravehicular activity,said device comprising: a first input (120A) for first data (ESD)indicative of a detection space around the individual; a second input(120B) for second data (HPD) indicative of a pose of the head (20) ofthe individual relative to a helmet (10), which is worn over the head(20) of the individual with spacing to allow the head (20) to moveinside the helmet (10); and processor circuitry (101) configured to:obtain the first data (ESD) on the first input (120A); obtain the seconddata (HPD) on the second input (120B); process the first data (ESD) todetect an object (40) of relevance to the individual in the detectionspace and determine a first position of the object (40) with respect tothe helmet (10); determine, based on the first position and the seconddata (HPD), a second position of the object (40) with respect to thehead (20), and cause an array of speakers (33) in the helmet (10) togenerate an audible sound inside the helmet (30) with a spatial origingiven by the second position.

C2. The device of C1, wherein the processor circuitry (101) isconfigured to cause the array of speakers (33) to generate the audiblesound with a discriminable characteristic that is indicative of theobject (40).

C3. The device of C2, wherein the discriminable characteristic is anaudible property of the audible sound other than the spatial origin.

C4. The device of C2 or C3, wherein the processor circuitry (101) isconfigured to process the first data (ESD) for detection of a hazardassociated with the object (40), and wherein the discriminablecharacteristic is indicative of the hazard.

C5. The device of C4, wherein the processor circuitry (101) is furtherconfigured to process the first data (ESD) for classification of thehazard into a hazard category among a plurality of a hazard categories,wherein the discriminable characteristic is indicative of the hazardcategory and differs between the hazard categories.

C6. The device of any one of C2-C5, wherein the processor circuitry(101) is configured to process the first data (ESD) for classificationof the object (40) into an object category among a plurality of objectcategories, wherein the discriminable characteristic is indicative ofthe object category and differs between the object categories.

C7. The device of any one of C2-C6, wherein the processor circuitry(101) is configured to modify the discriminable characteristic as afunction of a distance between the object (40) and the individual oranother individual in the detection space and/or a movement direction ofthe object (40) in relation to the individual or said another individualin the detection space.

C8. The device of any one of C2-C7, wherein the processor circuitry(101) is configured to modify the discriminable characteristic as afunction of the pose of the head.

C9. The device of any one of C2-C8, wherein the processor circuitry(101) is configured to modify the discriminable characteristic independence of a relation between the object (40) and a field of visionof the individual.

C10. The device of C9, wherein the processor circuitry (101) isconfigured to calculate the field of vision of the individual through avisor (12) of the helmet (10) based on the second data (HPD), anddetermine the relation between the object (40) and the field of vision.

C11. The device of C9 or C10, wherein the processor circuitry (101) isconfigured to generate the audible sound with different discriminablecharacteristics if the object (40) is within or outside the field ofvision.

C12. The device of any one of C2-C11, wherein the processor circuitry(101) is configured to estimate, based on the first data, a trajectory(T1) of the object, and to modify the discriminable characteristic as afunction of the trajectory (T1).

C13. The device of any preceding clause, wherein the processor circuitry(101) is configured to determine, based on the first data (ESD), awaypoint (W1) with respect to the helmet (10) on a path (P1) from afirst location of the individual to a second location (EP), determine aposition of the waypoint (W1) with respect to the head based on thesecond data (HPD), and generate a further audible sound with a spatialorigin given by the position of the waypoint (W1) with respect to thehead.

C14. The device of C13, wherein the processor circuitry (101) is furtherconfigured to cause the array of speakers (33) to verbalize the path(P1).

C15. The device of any preceding clause, wherein the processor circuitry(101) is configured to cause the array of speakers (33) to verbalizeobject data representative of the object (40).

C16. The device of C15, wherein the object data comprises one or moreof: a characterization of the object (40), a distance to the object(40), a direction to the object (40), or a movement instruction to theindividual.

C17. The device of any preceding clause, further comprising at least oneof the helmet (10), the array of speakers (33), a sensor arrangement(108; 31) configured to generate the first data, or a head trackingdevice (32) configured to generate the second data.

C18. The device of C17, wherein the sensor arrangement (108; 31) isconfigured to detect one or more of a shape, a distance, a temperature,or a composition.

C19. A space suit for extravehicular activity, said space suit (2)comprising a device according to any preceding clause.

C20. A computer-implemented method for assisting an individual thatwears a helmet during extravehicular activity, the helmet allowing theindividual to move its head within the helmet, said computer-implementedmethod comprising: obtaining (141) first data indicative of a detectionspace around the individual; obtaining (141) second data indicative of apose of the head of the individual relative to the helmet; detecting(142), based on the first data, an object of relevance to the individualin the detection space; determining (143), based on the first data, afirst position of the object with respect to the helmet; determining(144), based on the first position and the second data, a secondposition of the object with respect to the head; and causing (145) anarray of speakers in the helmet to generate an audible sound with aspatial origin given by the second position.

C21. A computer-readable medium comprising instructions which, wheninstalled on a processor (401), causes the processor (401) to performthe method of C20.

2. Performance-Based Feedback During Activity in a Low-GravityEnvironment

This part of the disclosure relates to a technique of providingperformance support in a low-gravity environment to improve theperception of how a task is performed by a user in the low-gravityenvironment. This part of the disclosure is particularly, but notexclusively, related to an inventive concept of evaluating measured bodyposes and gaze directions of the user for detection of deviations in theperformance of the task by the user and providing related feedback tothe user or to another user. The inventive concept may be implemented toprovide the feedback in real-time to the user that performs the task,thereby allowing the user to instantly correct any deviations.Alternatively, the feedback may be provided to the user after taskcompletion, so that the user is made aware of the deviations and cantake corrective measures next time the task is performed. Alternativelyor additionally, the feedback may be provided to another user. Real-timefeedback allows the other user to provide corrective instructions inreal-time to the user. In some embodiments, the low-gravity environmentis extraterrestrial and the activity an EVA. In other embodiments, theinventive concept is applied in preparation of EVA, for example duringrehearsal of tasks in a low-gravity environment on Earth. In both cases,the inventive concept has the technical effect of reducing the risk thatthe user inadvertently deviates from a rehearsed movement during EVA.Depending on implementation, the feedback may be audible/visual or inanother format.

FIG. 9 illustrates an individual or user 1 that performs a task inspace, or in a low-gravity environment in preparation of a spacemission. Here, the task is climbing down or up a ladder 160. It is to benoted that the term “task” is broadly referring to any sequence ofactions that are predefined to be performed by an individual in alow-gravity environment. Other non-limiting examples of tasks includeclimbing out of or into a hatch, entering or leaving a vehicle, jumping,lifting, performing repair or service of different types, etc. The taskmay be predefined in terms of body pose and gaze direction for thedifferent actions included in the task.

In FIG. 9 , the gaze direction is represented by an arrow GD, and thebody pose is represented by two body angles ⊖1, ⊖2, which are definedbetween limbs that meet at a joint or “keypoint” KP1, KP2. It isrealized that the body pose, depending on the task, may be defined by aset of body angles, and that the set of body angles may differ betweentasks, for example depending on the limbs that are relevant to the task.The gaze direction may be given in relation to the space suit 2, forexample the helmet 10. In some embodiments, the gaze direction isdefined by a set of gaze angles in relation to a coordinate systemassociated with the space suit 2, for example azimuth and elevationangles.

The task of descending the ladder 160 may be defined to include severalactions or movements, such as “lift right foot from step”, “lower rightleg”, “place right foot on step”, “lift left foot from step”, “lowerleft leg” and “place left foot on step”. These actions may be repeatedduring the task and may take any time to perform. The task is associatedwith a nominal performance scheme, NPS, which defines allowable bodyposes and gaze directions that are to be attained by the individual whenperforming the task. As will be described below, the NPS may takedifferent forms and is used for evaluating if the individual performsthe task with or without deviations.

FIG. 10 shows a device 220 which is configured to provide performancesupport in a low-gravity environment in accordance with the inventiveconcept. The device 220 is part of a system 200, which also includes afirst measurement arrangement 201, a second measurement arrangement 202,and a feedback device 203. The first measurement arrangement 201 isconfigured to generate first data indicative of the body pose of a user.The first data is denoted body pose data, BPD, in the following. Thefirst measurement arrangement 201 may comprise one or more body posesensors, for example as discussed with reference to the sensorarrangement 107 in FIG. 2 . The second measurement arrangement 202 isconfigured to generate second data indicative of the gaze direction ofthe user. The second data is denoted gaze direction data, GDD, in thefollowing. The second measurement arrangement 202 may comprise a gazetracking device. Alternatively, gaze direction may be approximated bythe orientation of the user’s head. Thus, in some embodiments, thesecond measurement arrangement 202 comprises a head tracking device, andthe GDD is the output of the head tracking device (cf. HPD in Chapter 1and Chapter 3). The gaze or head tracking device may be configured asdiscussed with reference to the sensor arrangement 107 in FIG. 2 . Thefeedback device 203 is configured to present feedback based on feedbackdata, FD, provided by the device 220. The feedback may be presented invisual and/or audible form. Audible feedback data may be presented inverbalized form or as dedicated signals indicative of a detecteddeviation. Visual feedback data may, for example, be presented toindicate the location and magnitude of a detected deviation on a graphicrepresentation of the user and/or to show the performed movement inrelation to an ideal movement. The feedback may also comprise one ormore calculated metrics related to the user’s performance of the task.

As an alternative or supplement to visual and audible feedback, anyother form of feedback may be presented by the feedback device 203. Forexample, haptic feedback may be given through actuators attached tolimbs or body parts of the user. The haptic feedback may be given byapplying forces, vibrations or motions to the user. The haptic feedbackmay be given to indicate the location and magnitude of the deviation. Insome embodiments, the actuators may be operated to provide the feedbackby inhibiting or counteracting the motion of a limb or body part foundto deviate. This type of haptic feedback may be seen as a “forcefeedback”. It is to be understood that “present feedback” is used hereinto broadly cover any way of conveying feedback to an individual.

As understood from the foregoing, the feedback device 203 may bearranged to present the feedback to the user that performs the task andmay thus be arranged in the space suit 2. Such a feedback device 203 maycomprise the display device 115 and/or the speaker system 114 in FIG. 2, and/or the above-mentioned actuators. Alternatively, the feedbackdevice 203 may be arranged to present the feedback to another user, whomay be in another space suit, in a spacecraft, a mission control center,a training facility, etc. Such a feedback device 203 may comprise adisplay device and/or a speaker system of any suitable type.

The device 220 comprises processor circuitry 101, which may correspondto the control device 101 in FIG. 2 , a first input device 220Aconfigured to receive the BPD from the first measurement arrangement201, and a second input device 220B configured to receive the GDD fromthe second measurement arrangement 202. The input devices 220A, 220B(“inputs”) may comprise any conventional hardware interface, optionallycombined with software, for receiving input signals. The device 220further comprises an output device 220C (“output”) configured to provideFD to the feedback device 203.

The device 220 in FIG. 10 may be included in the space suit 2, forexample by manufacture or retrofitting. The device 220 may be a unitarycomponent or an assembly of separate parts. It is also conceivable thatthe device 220 includes one or more additional components of the system200, for example the first measurement arrangement 201, the secondmeasurement arrangement 202, or the feedback device 203.

FIG. 11 is a flowchart of an example method 210 of providing performancesupport in a low-gravity environment. The method 210 iscomputer-implemented and may be performed by the device 220 in FIG. 10 .The method 210 may be implemented by hardware or a combination ofhardware and software, for example by program instructions stored in amemory in the device 220.

In step 211, BPD is obtained via the first input 220A. Depending onimplementation, the BPD may include raw sensor data from sensor(s) inthe first measurement arrangement 201, pre-processed data, or datarepresentative of the body pose, such as a first set of body angles.

In step 212, GDD is obtained via the second input 220B. Depending onimplementation, the GDD may include raw sensor data from sensor(s) inthe second measurement arrangement 202, pre-processed data, or datarepresentative of the gaze direction, such as a second set of gazeangles.

In step 213, a first time series of body poses is determined based onthe BPD, and second time series of gaze directions are determined basedon the GDD. The first and second time series are determined to representthe individual while performing a task. Depending on the content andformat of BPD and GDD, step 213 may comprise reading data representativeof body pose and/or gaze direction from BPD and/or GDD, or processingBPD and/or GDD for determination of such data.

The first time series comprises a time sequence of body poses andrepresents the momentary body pose at different time points over a firsttime period. Correspondingly, the second time series comprises a timesequence of gaze directions and represents the momentary gaze directionat different time points over a second time period. The first and secondtime periods may or may not be overlapping, and the time points in thefirst and second time series may or may not coincide. In someembodiments, however, the first and second time series are coordinatedso that there is an approximate temporal match between body poses andgaze directions.

In step 214, the above-mentioned nominal performance scheme, NPS, isobtained. The NPS may be obtained from a memory in the device 220, orfrom another storage device in the system 200. In some embodiments, theNPS is specific to a task and may be obtained based on a taskidentification performed in step 213. In some embodiments, the NPSdefines an ideal sequence of body poses and gaze directions to beattained during performance of a respective task. Another format of theNPS will be described below with reference to FIGS. 12-13 . The NPS mayalso define allowable deviations for the body poses and gaze directions,and the allowable deviations may differ between different stages of thetask. Further, the allowable deviations may differ between differentparts of the body pose, for example between different body angles.

In step 215, the first and second time series are evaluated in relationto the NPS for detection of a performance deviation. The implementationof step 215 depends on the format of the NPS, and also on the requiredtime response of the method 210 to provide the FD, but may be seen toinvolve a comparison between a respective body pose in the first timeseries to a corresponding body pose defined by the NPS, and a comparisonof a respective gaze direction in the second time series to acorresponding gaze direction defined by the NPS, and a combining of theresults from the respective comparison.

In step 216, the FD is generated to be representative of the outcome ofthe evaluation in step 215, and output for presentation by the feedbackdevice 203. In some embodiments, step 216 comprises generating controlsignals for operating the feedback device 203. In other embodiments,step 216 comprises transmitting the FD in a format adapted to thefeedback device 203. In yet other embodiments, step 216 comprisestransmitting the FD to the feedback device, which independentlygenerates the appropriate control signals.

If the method 210 is performed to provide the FD after completion of thetask, processing efficiency may be less of an issue and the method 210may be implemented to consume any amount of memory resources andprocessing capacity of the device 220. On the other hand, if the method201 is performed to provide the FD in real-time, processing and powerefficiency may be of essence. Further, for real-time feedback, themethod 210 has to be implemented to produce the results of theevaluation in step 215 in synchronization with the progression of thetask.

FIG. 12 is a block diagram of an example processing system 230 which isconfigured to implement the method 210 and is operable to providereal-time feedback. However, the processing system 230 may also beoperated to provide feedback after task completion. The processingsystem 230 may be included in the device 220, for example as part of theprocessor circuitry 101. The processing system 230 comprises blocks ormodules 231-235.

Module 231 implements step 213 (FIG. 11 ) and is configured to receivethe BPD and GDD and repeatedly generate the first and second time seriesfor a predefined time period or time window which is a subset of thetotal time required to perform the task. The time window may also besmaller than the duration of a respective action included in the task.The length of the time window is a compromise between processingefficiency and performance and may be determined by testing or modeling.For example, the time window may be set in the range of 10 ms - 10 s, orin the range of 100 ms - 5 s. Consecutive time windows may beoverlapping or non-overlapping. The resulting first and second timeseries are designated [BP] and [GD] in FIG. 12 . As shown in FIG. 12 ,module 231 may also output a momentary body pose, BP′, and a momentarygaze direction, GD′, for use by module 234, as described below.

Module 232 implements part of step 215 (FIG. 11 ) and is configured toreceive and process the first and second times series, [BP], [GD] fordetermination of action data. The action data comprises at least onepredefined action Aij and optionally an associated probability Pij. Asshown, module 232 may comprise a trained machine learning-based module232′ (MLM_(T)) which is configured to map incoming [BP], [GD] topredefined actions that may be part of the tasks to be performed by theuser. The MLM_(T) 232′ may be a conventional activity recognitionalgorithm and may utilize a neural network for action classification.Depending on implementation, as is well-known to the skilled person, theMLM_(T) 232′ may output a single action, optionally together withprobability value, or a plurality of actions and a probability value ofeach action. The respective probability value represents the likelihoodthat the input data [BP], [GD] corresponds to the associated action.

Module 233 implements part of step 215 and is configured to evaluate theat least one action Aij and/or the at least one probability Pij frommodule 232 in relation to a validation criterion, which is based on theNPS. In the illustrated example, the NPS comprises an action sequencedefinition, ASD, which is used to define the validation criterion.Examples of validation criteria and use of ASD will be described belowwith reference to FIGS. 13A-13B. Module 233 is further configured toselectively, if the validation criterion is violated, activate module234.

Module 234 implements part of step 215 and is configured to perform adeviation analysis in relation to reference data, RD, included in theNPS. In some embodiments, the deviation analysis comprises evaluating amomentary body pose (BP′) and/or a momentary gaze direction (GD′),provided by module 231, in relation to the RD. Module 234 may beconfigured to obtain BP′ and/or GD′ from module 231 on demand. Examplesof the deviation analysis and RD are given below with reference to FIG.13C and FIGS. 14B-14C.

Module 235 implements step 216 and is configured to generate FD based onthe outcome of the deviation analysis performed by module 234.

It should be noted that modules 232, 233 are operated to repeatedlygenerate and evaluate [BP], [GC] at consecutive time points, that module234 is operated to perform the deviation analysis when motivated by theaction data, and that module 235 is operated to generate the FD based onthe outcome of the deviation analysis. Thereby, it is possible togenerate real-time feedback. Further, the deviation analysis is onlyperformed when the validation criterion is violated, which will improveprocessing and power efficiency.

FIG. 13B is a flow chart of an example procedure that may be part ofevaluation step 215 (FIG. 11 ). The example procedure in FIG. 13B may beperformed by the processing system 230 in FIG. 12 .

In step 251, which may be performed by module 232, action data isdetermined for the first time series [BP] and/or the second time series[GD]. Thus, although module 232 is illustrated in FIG. 12 to receiveboth [BP] and [GD], it is conceivable that the action data is determinedbased on either [BP] or [GD].

Step 252 may be performed by module 233 and comprises an evaluation ofthe one or more actions Aij in the action data in relation to avalidation criterion, which is based on the action sequence definition,ASD. The ASD is predefined and defines allowable actions to be performedby the user within a task and the allowable sequencing or ordering ofthe allowable actions. A graphical example of an ASD is shown in FIG.13A. In the illustrated example, the ASD defines actions for two tasksT1, T2. Task T1 comprises allowable actions A11-A17, and task T2comprises allowable actions A21-A26. Solid arrows 240 indicate allowableswitches between consecutive actions of a task, dashed arrow 241indicates an allowed jump to skip one or more actions of a task, anddashed arrow 242 indicates an allowed jump from an action in one task toan action in another task. The ASD thus defines a predefined sequence ofactions for one or more tasks. However, as seen, the ASD is not limitedto a linear sequence of actions, but may incorporate loops, forward andbackwards steps, and skipping jumps both between actions within a taskand between tasks. The ASD may be seen as a graph, which is defined byactions and links between actions, with the links defining one or moreactions (“expected actions”) to follow upon the respective action at thenext time step. It is to be noted that, although not shown in FIG. 13A,it may be allowed for each action to be followed by the same action.This will make the evaluation in step 251 independent of how fast theuser performs the task.

In one example, T2 is the task of climbing the ladder 160 in FIG. 9 ,and A21-A26 represent the above-mentioned actions: “lift right foot fromstep” (A21), “lower right leg” (A22), “place right foot on step” (A23),“lift left foot from step” (A24), “lower left leg” (A25), and “placeleft foot on step” (A26). The ASD in FIG. 13A allows the user to repeatA21-A26 and allows transitions back and forth between adjacent actions.

In some embodiments of step 252, the validation criterion is violated ifan evaluated action in the action data from step 252 deviates from theexpected action(s) according to the ASD. An expected action isidentified in the ASD in relation to the evaluated action in a precedingexecution of step 252, for example the last execution. If the actiondata includes a single action, the evaluated action is the singleaction. If the action data includes plural actions, the evaluated actionmay be the action that is associated with the highest probability. It isalso conceivable that there are plural evaluated actions, which may beall actions in the action data that have a probability above aprobability limit.

An example of the validation by step 252 is shown in FIG. 14A. Here,each dot represents an action determined by step 251 and evaluated bystep 252 at consecutive time steps, i.e. for consecutive time series[BP], [GD]. As seen, the user is deemed to perform A11 for a number oftime steps and then the user appears to switch to A12, which is allowedaccording to the ASD in FIG. 13A. At time point tc, the user appears toswitch to action A17, which is not allowed according to the ASD in FIG.13A. Thus, at time tc, step 252 will detect a violation of thevalidation criterion.

Step 253 may also be performed by module 233 and may supplement orreplace step 252. Step 253 comprises an evaluation of one or moreprobability values Pij in the action data in relation to a validationcriterion. If the action comprises plural probability values, themaximum probability value is evaluated. For example, the validationcriterion may be violated if the probability is below a reference value,which may be given by the NPS.

In step 254, which may be performed by module 233, the outcome of step252 and/or step 253 is evaluated. Depending on implementation, step 254may detect a need for deviation analysis if a violation is determined byat least one of steps 252, 253 (OR criterion), or by both steps 252, 253(AND criterion). If a need for deviation analysis is determined, theprocedure continues to step 255. Otherwise, the procedure may proceed tostep 213, which determines new first and second time series.Alternatively, the procedure may proceed to step 216 and cause thefeedback device 203 to indicate absence of deviations.

Step 255 may be performed by module 234 and comprises theabove-mentioned deviation analysis in relation to the above-mentionedreference data, RD, for detection of performance deviation(s).

If a performance deviation is detected in step 255, step 256 directs theprocedure to step 257. Otherwise, the step 256 directs the procedure tostep 213, which determines new first and second time series.Alternatively, step 256 may direct the procedure step 216, which maycause the feedback device 203 to indicate an absence of deviations.

In step 257, which may be performed by module 234, the origin of theperformance deviation(s) is identified, in the form of one or morenon-conforming body parts and/or a non-conforming gaze direction. Step257 may also quantify the performance deviation, by calculating one ormore metrics based on the outcome of the deviation analysis in step 255.The output of step 257 may then be included in the FD by step 216.

FIG. 13C is a flow chart of an example procedure that may be part of thedeviation analysis in step 255 of FIG. 13B. As understood from FIG. 13B,step 255 is performed at a specific time point, subject to detection ofviolation(s) by steps 252-253. In the example of FIG. 14A, this timepoint corresponds to time tc.

The example procedure in FIG. 13C presumes that the reference data RDassociates a respective action in the ASD with one or more allowableranges for body pose and/or gaze direction. Thus, RD may define one ormore allowable ranges for the gaze direction, for example an allowableelevation angle range and/or an allowable azimuth angle range.Alternatively or additionally, the RD may define one or more allowableranges for the body pose, for example an allowable orientation range foreach body part deemed to be of relevance to the action. In someembodiments, RD may define a body angle range for each joint or keypoint(cf. FIG. 9 ) deemed to be of relevance to the action. It should benoted that the allowable range may differ between joints.

Step 261 obtains momentary data of body pose and/or gaze direction atthe time point when the violation is detected. This corresponds tomodule 234 being operated to retrieve BP′, GD′ from module 231 in FIG.12 .

Step 262 comprises evaluating the momentary data in relation to thecorresponding allowable range(s) given by RD. A performance deviationmay be detected if the momentary data falls outside one or moreallowable ranges. Step 262 may select the allowable range(s) to be usedin the evaluation from RD. In some embodiments, each such allowablerange is given by the last action that was performed by the user beforethe violation was identified, i.e. the latest non-violated action. Withreference to FIG. 14A, step 262 would select the allowable range(s) isextracted for action A12 in RD.

It should be realized that the example procedure in FIG. 13C provides aprocessing efficient technique for detecting performance deviations.

FIG. 14B is a graph of a body angle ⊖n as a function of time, and FIG.14C is a corresponding graph of gaze direction as a function of time.The curve 281 is included in a first time series, [BP], and the curve281 is included in a second time series, [GD], and Δt represents theabove-mentioned time window. Further, Δ⊖n represents the allowable rangeof the body angle en, and ΔGD represents the allowable range of the gazedirection. Further, 281′ designates the momentary body angle when theviolation is detected at time point tc, and 282′ designates themomentary gaze direction at time point tc. As seen in FIG. 14B, themomentary body angle 281′ falls outside Δ⊖n, which will cause step 262to identify a performance deviation for body angle ⊖n. On the otherhand, as seen in FIG. 14C, since the momentary gaze direction 282′ fallswithin ΔGD, no performance deviation will be identified for the gazedirection GD.

The allowable ranges may be set based on a theoretical model of themovement pattern of an individual wearing a space suit in a low-gravityenvironment. It is also conceivable to calculate the allowable rangesfrom BPD and GDD that are measured for one or more individuals whenperforming a respective action, preferably a large number of times andunder well-controlled conditions. For example, the allowable range for aparameter may be calculated as a function of the measured values of theparameter, for example to represent the dispersion of the measuredvalues. For example, the allowable range may be given as function of thestandard deviation or the interquartile range of the measured values. Itshould be realized that the device 220 may be used for collecting themeasured values by minor modification, for example by including astorage/calculation function that is activated when the device 220 isset in a calibration mode. For example, the processing system 230 inFIG. 12 may be supplemented by a storage module, which is operable tojointly store associated samples of input data and output data of module232, for example [BP] and/or [GD] and a resulting action. Alternativelyor additionally, the processing system 230 may be supplemented by acalculation module configured to calculate and set the allowable rangesbased on the associated samples of input data and output data of module232.

FIG. 13D is a flow chart of a procedure that may be part of step 216(FIG. 11 ). The procedure may comprise all of steps 271-274, or any oneof step 271, step 272 or steps 273-274.

Step 271 comprises including an indicator, in the FD, of thenon-confirming body part and/or non-conforming gaze direction, forexample as identified in step 257. Step 272 comprises providingcorrective feedback as part of the FD. The corrective feedback mayinclude instructions on corrective measures to be taken by the user toremedy the performance deviation(s). Step 271 and/or step 272 will helpthe user to adhere to the NPS.

Step 273 comprises identifying or determining a next action for the userto perform. The next action is identified is the ASD as a subsequentaction in relation to a current non-violated action. For example, thenext action may be given by a solid arrow in the example of FIG. 13A andmay correspond to the most plausible action to be taken by the userafter a current action. In step 274, the feedback device 203 is causedto present the next action. The combination of steps 273-274 will guidethe user to perform the actions of a task in a predefined order.

One advantage of some embodiments described in the foregoing is that theuser may be warned whenever the user performs incorrect movements aspart of a task.

Another advantage of some embodiments is that the user may be givensupport and advice on what movement and/or part of movement is wrong andhow to correct it.

Further, the technique described herein may be used during training orrehearsal to verify if an individual has learned to perform a task inthe manner required.

In the following, clauses are recited to summarize some aspects andembodiments as disclosed in the foregoing.

C1. A device for performance support in a low-gravity environment, saiddevice comprising: a first input (220A) for first data (BPD) indicativeof a body pose of an individual; a second input (220B) for second data(GDD) indicative of a gaze direction of the individual; and processorcircuitry (101) configured to: obtain the first data (BPD) on the firstinput (220A); obtain the second data (GDD) on the second input (220B);determine, based on the first data (BPD) and the second data (GDD), afirst time series of body poses and a second time series of gazedirections that represent the individual performing a task; obtain anominal performance scheme (NPS) for the task; perform an evaluation ofthe first and second time series in relation to the nominal performancescheme (NPS) for detection of a performance deviation; and provide,based on the evaluation, feedback data (FD) for presentation by afeedback device (203).

C2. The device of C1, wherein the processor circuitry (101) isconfigured to determine the first and second time series for apredefined time period (Δt), which is a subset of a total time periodfor performing the task.

C3. The device of any preceding clause, wherein the task comprises apredefined sequence of actions (ASD), and wherein the processorcircuitry (101) is configured to, in the evaluation, evaluate the firstand/or second time series to determine action data (Aij, Pij) thatrelates the first and/or second time series to the predefined sequenceof actions (ASD), and selectively, depending on the action data (Aij,Pij), perform a deviation analysis for the detection of the performancedeviation.

C4. The device of C3, wherein the action data (Aij, Pij) comprises atleast one of: an action (Aij) deemed to correspond to the first and/orsecond time series, or a probability (Pij) that the action (Aij)corresponds to the first and/or second time series.

C5. The device of C4, wherein the processor circuitry (101) isconfigured to evaluate at least one of the action (Aij) or theprobability (Pij) in relation to a validation criterion, which is basedon the nominal performance scheme (NPS), and to selectively perform thedeviation analysis if the validation criterion is violated.

C6. The device of C5, wherein the nominal performance scheme (NPS)comprises a definition of the predefined sequence of actions (ASD).

C7. The device of C5 or C6, wherein the validation criterion is violatedif the action (Aij) deviates from one or more expected actions accordingto the predefined sequence of actions (ASD) and/or if the probability(Pij) is below a reference value.

C8. The device of any one of C5-C7, wherein the nominal performancescheme (NPS) comprises reference data (RD) that associates a respectiveaction among the predefined sequence of actions with one or moreallowable ranges for the body pose and/or the gaze direction of theindividual, wherein the processor circuitry (101), in the deviationanalysis, is configured to obtain momentary data (BP′, GD′) for the bodypose and/or the gaze direction at a time point (tc) when the validationcriterion is violated, and evaluate the momentary data (BP′, GD′) inrelation to the reference data (RD) for detection of the performancedeviation.

C9. The device of C8, wherein the momentary data comprises momentaryorientations (BP′) of a plurality of body parts, wherein the referencedata (RD) associates the plurality of body parts with allowable rangesof orientation, and wherein the processor circuitry (101) is configuredto detect the performance deviation as a deviation of a momentaryorientation of a body part in relation to an allowable range (Δ⊖n) oforientation for the body part, the allowable range (Δ⊖n) of orientationbeing given by the reference data (RD).

C10. The device of C9, wherein the momentary orientation of the bodypart comprises an angle (⊖n) of the body part in relation to anotherbody part.

C11. The device of any one of C8-C10, wherein the momentary datacomprises a momentary gaze direction (GD′), and wherein the processorcircuitry (101) is configured to detect the performance deviation as adeviation of the momentary gaze direction (GD′) in relation to anallowable range (ΔGD) for the gaze direction, the allowable range (ΔGD)for the gaze direction being given by the reference data (RD).

C12. The device of any one of C3-C11, wherein the processor circuitry(101) is configured to determine, among the predefined sequence ofactions (ASD), a subsequent action in relation to a current action, andcause the feedback device (203) to present the subsequent action.

C13. The device of any one of C3-C12, wherein the processor circuitry(101) comprises a trained machine learning-based model (232′) which isconfigured to operate on the first and/or second time series fordetermination of the action data (Aij, Pij).

C14. The device of any preceding clause, wherein the processor circuitry(101) is further configured to identify at least one of a non-conformingbody part or a non-conforming gaze direction associated with theperformance deviation.

C15. The device of C14, wherein the feedback data (FD) is arranged toindicate said at least one of a non-conforming body part or anon-conforming gaze direction, and optionally to provide correctiveinstructions in relation to said at least one of a non-conforming bodypart or a non-conforming gaze direction.

C16. The device of any preceding clause, further comprising at least oneof a first measurement arrangement (201) configured to generate thefirst data, a second measurement arrangement (202) configured togenerate the second data, or the feedback device (203).

C17. A space suit for extravehicular activity, said space suit (2)comprising a device according to any preceding clause.

C18. A computer-implemented method of providing performance support in alow-gravity environment, said computer-implemented method comprising:obtaining (211) first data (BPD) indicative of a body pose of anindividual from a first measurement arrangement (201); obtaining (212)second data (GDD) indicative of a gaze direction of the individual froma second measurement arrangement (202); determining (213), based on thefirst data (BPD) and the second data (GDD), a first time series of bodyposes and a second time series of gaze directions that represent theindividual performing a task; obtaining (214) a nominal performancescheme (NPS) for the task; performing (215) an evaluation of the firstand second time series in relation to the nominal performance scheme(NPS) for detection of a performance deviation; and causing (216) afeedback device (203) to present feedback data (FD) representing aresult of the evaluation.

C19. A computer-readable medium comprising instructions which, wheninstalled on a processor (401), causes the processor (401) to performthe method of C18.

3. Illumination-Based Assistance During Extravehicular Activity

This part of the disclosure relates to a technique of providingassistance to one or more individuals or users during EAV. This part ofthe disclosure is particularly, but not exclusively, related to aninventive concept of operating a computer device to process a set ofrules to identify relevant user instructions in relation to one or moreobjects detected in the surroundings of the user(s) and provide the userinstructions to the user(s) by selective projection of light in relationthe object(s). The user instructions thereby provide active guidance tothe individual, for example in relation to a task to be performed. Theinventive concept reduces the reliance on cognitive processing by theuser(s) to take decisions on how to proceed in a situation, and insteadthe cognitive processing of information about the surroundings inrelation to the task at hand is offloaded to the computer device. Thiswill serve to reduce the cognitive load on the user(s) and allow theuser(s) to focus on implementing the task as well as possible. It mayalso improve the safety of the user(s), as well as user compliance withsafety protocols and other plans, procedures and schemes that may bedefined for individuals in space. The selective projection of lightprovides a convenient way of conveying the user instructions to theuser(s) since it is independent of conventional audible communicationand allows the user instructions to be distributed to any number ofusers. The illumination-based assistance according to the inventiveconcept is also intuitive to the user and may be provided withoutoccluding or obstructing the user’s field of vision.

FIG. 15 illustrates an individual or user 1 that performs a task inspace. An illumination arrangement 301 is attached to the space suit 2,in this example on the helmet 10. The illumination arrangement 301 isoperable to project visible light 301′ into the surroundings of theuser. In the example of FIG. 15 , a monitoring sensor 31 is alsoarranged on the helmet 10 to generate sensor data that is indicative ofa detection space outside and around the space suit 2. In accordancewith the above-mentioned inventive concept, the illumination arrangement301 is operated to convey user instructions to the user 1, or to anotheruser (not shown), in relation to objects that are detected based on thesensor data from the monitoring sensor 31.

FIG. 16 shows a device 320 which is configured to provideillumination-based assistance in a low-gravity environment. The device320 is part of a system 300, which also includes one or more monitoringsensors 31 (one shown), and the illumination arrangement 301.Optionally, the system 300 may also comprise a measurement arrangement201 configured to generate first data indicative of the body pose of auser and/or a head tracking device 32 configured to generate second dataindicative of the head pose of the user.

The monitoring sensor(s) 31 may be the same or similar as described inChapter 1 above. External sensor data, ESD, which is generated by themonitoring sensor(s) 31, is indicative of objects in a detection spacein relation to the monitoring sensor(s) 31. The detection space isdefined by the volume in which the one or more monitoring sensors 31 areresponsive to objects. The measurement data may be indicative of theposition of an object and/or one or more other properties of an object,such as its shape, its temperature, its color, its composition, etc. Themonitoring sensor 31 may be part of the arrangement 108 as describedwith reference to FIG. 2 and may be implemented as a vision sensor.

The illumination arrangement 301 is configured to generate visible light301′ (FIG. 15 ) based on illumination control data, ICD, provided by thedevice 320. In some embodiments, as shown in FIG. 16 , the illuminationarrangement 301 comprises at least part of the light control circuitry116 and the light source 117 as described with reference to FIG. 2 .

As noted above, the inventive concept involves a selective projection ofvisible light. In this context, “selective projection” implies that thevisible light is spatially controlled to convey a user instruction inrelation to one or more objects. In one example, the visible light 301′is spatially confined and selectively directed to a region on or closeto an object. For example, the spatially confined light may be acollimated laser beam or an appropriately focused light beam, which isdirected onto the region. Such a laser beam or focused light beam may ormay not be spatially controlled to generate an image in the region. Theimage may convey the user instruction and may comprise one or moresymbols and/or plain text. In another example, a static beam of light isprojected within the detection space to illuminate the region,optionally by providing the above-mentioned image. In some embodiments,the illumination arrangement 301 comprises a laser projector or a videoprojector. The video projector may be a so-called short throw projector,which is capable of projecting an image at a distance of 1-10 meters.Depending on implementation, the illumination arrangement 301 may beoperable to generate the visible light 301′ with different visual cues,such as different images, colors, radiant intensities, etc.

As noted, the measurement arrangement 201 and the head tracking device32 are optional. Like in Chapter 2, the first data generated by themeasurement arrangement 201 is denoted body pose data, BPD. Themeasurement arrangement 201 may comprise one or more body pose sensors,for example as discussed with reference to the sensor arrangement 107 inFIG. 2 . Like in Chapter 1, the second data generated by the headtracking device 32 is denoted head pose data, HPD. As shown, the headtracking device 32 may be included in the sensor arrangement 107.

The device 320 comprises processor circuitry 101, which may correspondto the control device 101 in FIG. 2 , a first input device 320Aconfigured to receive the ESD from the monitoring sensor(s) 31, a second(optional) input device 320B configured to receive BPD from themeasurement arrangement 201, and third (optional) input device 320Cconfigured to receive HPD from the head tracking device 322. The inputdevices 320A, 320B, 320C (“inputs”) may comprise any conventionalhardware interface, optionally combined with software, for receivinginput signals. The device 320 further comprises an output device 320D(“output”) configured to provide illumination control data, ICD, to theillumination arrangement 301.

In some embodiments, the device 320 in FIG. 16 is included in the spacesuit 2, for example by manufacture or retrofitting. In otherembodiments, the device 320 is separate from the space suit 2. Forexample, the device 320 may be included in a vehicle, such as aspacecraft or a space exploration vehicle.

Likewise, the monitoring sensor(s) 31 and the illumination arrangement301 may be included in or attached to the space suit 2, as shown in FIG.15 and described hereinabove. However, in variants, the monitoringsensor(s) 31 and/or the illumination arrangement 301 is separate fromthe space suit 2. Thus, the system 300 may be fully integrated with thespace suit 2, fully separated from the space suit 2, or anything inbetween.

The device 320 may be a unitary component or an assembly of separateparts. It is also conceivable that the device 320 includes one or moreadditional components of the system 300, for example the monitoringsensor(s) 31, the measurement arrangement 201, the head tracking device32, or the illumination device 301.

FIG. 17 is a flow chart of an example method 310 for providingassistance during EVA. The method 310 is computer-implemented and may beperformed by the device 320 in FIG. 16 . The method 310 may beimplemented by hardware or a combination of hardware and software, forexample by program instructions stored in a memory in the device 320.

In step 311, first data is obtained via the first input 320A. The firstdata corresponds to the ESD in FIG. 16 . The ESD may include raw sensordata from the monitoring sensor(s) 31, pre-processed data, or anindication of detected objects, optionally together with an indicationof one or more object properties.

In step 312, the first data (ESD) is processed for obtaining one or moreobject properties for a respective object. In this context, processingof the ESD may range from a full processing of raw sensor data, forexample in accordance with step 142A in FIG. 7A, to an extraction ofindications of object properties from the ESD. Irrespective ofimplementation, step 312 produces one or more object properties for arespective object detected by the monitoring sensor(s) 31. It may benoted that the monitoring sensor(s) 31 and/or the processing in step 312may be targeted to specific objects of relevance, and that the objectproperties may be provided only for such objects of relevance. The oneor more object properties may comprise one or more of position, shape,temperature, color, composition, speed, movement direction,acceleration, etc.

As indicated in FIG. 17 , step 312 may comprise a step 312A ofdetermining a classification of the object(s) into one or more objectcategories. Again, the classification may be obtained in step 312A byanything from a full processing of raw sensor data, for example inaccordance with step 142B in FIG. 7A, to an extraction of one or moreindications of object category from the ESD. As used herein, objectcategory is considered to be an object property. The use of objectcategories may simplify the evaluation in step 313 (below). Non-limitingexamples of object categories include “rock”, “sharp object”, “unstableground”, “stable ground”, “hot object”, “cold object”, “tool”, “toolcategory”, “other user”, “vehicle”, “vehicle category”, “manipulatedobject”, “anchoring point”, etc.

Step 313 comprises evaluating the one or more object properties inrelation to a rule definition to determine one or more userinstructions. The rule definition comprises a set of rules that definesa dedicated processing to be performed based on the one or more objectproperties. The rule definition may also comprise criteria for selectingspecific user instructions based on the result of the dedicatedprocessing. Each rule may thus be seen to define one or more criteria tobe fulfilled by the one or more object properties in order for aspecific user instruction to be selected. Examples of rules andresulting user instructions are presented below with reference to FIGS.19-20 .

In step 314, the illumination arrangement 301 is caused to provide theuser instruction(s) determined by step 313 by selective projection oflight in relation to one or more objects among the detected objects. Insome embodiments, step 314 comprises generating control signals foroperating the illumination arrangement 301 to perform the selectiveprojection of light. In other embodiments, step 314 comprises providinghigh-level commands to the illumination arrangement 301, for examplecomprising one or more positions for the selective projection, andoptionally an extent of the projected light.

In some embodiments, step 314 causes the illumination arrangement 301 togenerate the light with a visual cue. In some embodiments, step 314causes the illumination arrangement 301 to represent different userinstructions with different visual cues, for example depending onurgency, risk level, type of user instructions, object property, rule,etc.

FIG. 18 is a block diagram of an example processing system 330 which isconfigured to implement the method 310. The processing system 330 may beincluded in the device 320, for example as part of the processorcircuitry 101. The processing system 330 comprises blocks or modules331-333.

Module 331 implements step 312 (FIG. 17 ) and is configured to receiveand process the ESD for determination of object properties. The outputof module 331 is object property data OPD comprising on or more objectproperties for the respective object. In the illustrated example, module331 comprises a trained machine learning-based module 331′ (MLM_(T))which is configured to determine object properties, optionally includingobject categories. The MLM_(T) 331′ may be one or more conventionalalgorithms for object detection, for example as listed with reference toFIG. 7A in Chapter 1.

Module 332 implements step 313 and is configured to evaluate the OPDfrom module 331 for determination of user instruction(s) UI. The UI maybe output in the form of identifiers or indices of predefined userinstructions. The evaluation by module 332 is made in relation to theabove-mentioned rule definition, here represented as [R]. As indicated,module 332 may also operate on BPD received on input 320B and/or HPDreceived on input 320C (FIG. 16 ). An example of the use of BPD and HPDwill be given below with reference to FIG. 20E. In some embodiments,module 332 is configured to perform the evaluation by use of conditionalcontrol statements given by [R], for example IF statements or the like.The evaluation may for example be represented as a decision tree. Insome embodiments, the module 332 is configured to perform the evaluationby use of artificial intelligence (AI) or machine learning (ML), whichis configured to implement the rule definition [R].

Module 333 implements step 314 and configured to generate the ICD torepresent the UI generated by module 332. As understood from theforegoing, module 333 may be configured to generate the ICD as controlsignals or high-level commands.

FIG. 19 shows non-limiting examples of rules R1-R7 that may be includedin the rule definition [R]. R1 is a risk detection rule for identifyingat least one object that poses an actual or potential danger to a user.R2 is a task control rule for identifying an ordering among a pluralityof objects for use in performing a task. R3 is a completion rule foridentifying when a task is completed. R4 is a manipulation control rulefor identifying a manipulation operation to be performed on an object.R5 is a retrieval rule for identifying at least one object that is atrisk of leaving a field of vision and/or be beyond a reach limit of auser. R6 is a work sharing rule for selecting a user to perform anoperation. R7 is an anchor detection rule for identifying at least oneobject that is a safe anchoring point. As noted above, a rule may defineboth how to process object properties and what user instruction(s) toprovide.

Reverting to FIG. 18 , module 332 may comprise logic configured toautomatically identify the rule(s) to be evaluated based on the incomingOPD. Such logic may be seen to provide situational awareness and may beprovided by AI, machine learning, etc. Alternatively or additionally,the selection of rule(s) to be evaluated by module 332 in a particularsituation may be based on external input, for example by a useroperating a manual control (cf. 112 in FIG. 2 ) or by inputting a voicecommand to a voice control device connected to module 332.

Examples of the use of rules R1-R7 in FIG. 18 will now be given withreference to FIGS. 20A-20G. The examples are given for illustrativepurposes only and are not intended to limit the use of the respectiverule or to exclude the use of other rules. Although the illuminationarrangement 301 is attached to the helmet 10 of a user 1 in allexamples, it may instead be attached to another portion of the spacesuit 2, or be provided separately from the individual 1, for exampleattached to a supporting structure of a vehicle. Although not shown inFIGS. 20A-20G, the examples presume that the illumination arrangement301 is connected to the device 320, which is operated to control theselective projection of the visible light 301′.

In some embodiments, the device 320 is configured to cause theillumination arrangement 301, based on the risk detection rule R1, toprovide a user instruction to actively guide a user to avoid objectsthat are identified to pose an actual or potential danger to the user.These embodiments will improve the safety of users during EVA. In theexample of FIG. 20A, objects 340, 341 may represent different surfacehazards for a user 1 walking on an extraterrestrial surface. Forexample, the octagonal objects 340 may correspond to lose or slipperysurface material (“unstable surface”), and the pentagonal objects 341may represent jagged rocks or rock edges (“rock”, “sharp object”). Inaccordance with R1, the objects 340 may be detected based on shape,color or composition, and the objects 341 may be detected based onshape.

In one example, not shown, the beam 301′ may be directed onto theobjects 340, 341 to instruct the user to avoid the objects 340, 341.Different visual cues may be used to represent different risk levels,thereby presenting an ordering of risk to the user. Further, the objectsto be illuminated may be determined based on their distance to the user,so that the beam 301′ is preferentially directed onto the objectsclosest to the user.

In another example, as shown, the beam 301′ may be directed toilluminate a safe area 350 for the user 1, to instruct the user to moveto the illuminated area 350. By repeatedly performing the method 310(FIG. 17 ), the device 320 is operable to guide the user 1 along a safepath 360 in relation to the objects 340, 341 as the user 1 moves alongthe illuminated areas 350. The user may be guided on a correspondingsafe path when moving by gripping different objects, for example alongthe ladder 51 in FIG. 8B. Further, the safe path may be determined byconsidering the amount of gravity and the kinematic capabilities of theuser, for example given by suit constraints, human kinetics, field ofvision, etc.

In another example, R1 is configured to identify the objects to beavoided based on their temperature (“hot object”, “cold object”). A userinstruction not to touch the objects may be provided by guiding the beam301′ onto the objects to indicate that they pose a danger by theirtemperature.

In some embodiments, the device 320 is configured to cause theillumination arrangement 301, based on the task control rule R2, toprovide user instructions to actively guide the user 1 to use theobjects in a specific order. The rule R2 may also define the objectsproperties to be used to detect the different objects that are to beordered. The different objects may, for example, correspond to differenttools or materials that are used in a task. These embodiments willenhance user compliance with safety protocols, working protocols,service manuals, etc. In the example of FIG. 20B, the beam 301′ issequentially directed to illuminate objects 350 in the order they are tobe used, as indicated by encircled numbers 1, 2 and 3. The sequentialillumination may be made before the user 1 initiates the task and/or asthe user 1 completes various substeps of the task, for example toilluminate the next object to be used. Such sequential illumination ishighly intuitive to the user. To the extent that two or more objects areto be used in combination, these objects may be simultaneouslyilluminated as part of the sequential illumination. The sequentialillumination may also comprise projecting different images onto thedifferent objects, with the different images designating the order ofuse. Such an image may be an alphanumeric, for example a number as shownin FIG. 20B. Instead of sequential illumination, the different objectsmay be simultaneously illuminated with a respective image thatdesignates the order of use, for example an alphanumeric.

In some embodiments, the device 320 is configured to cause theillumination arrangement 301, based on the completion rule R3, toprovide a user instruction to actively guide a user to collect aplurality of objects when a task is completed. These embodiments willmitigate the risk that the user forgets tools or material in space aftertask completion. In the example of FIG. 20C, the illuminationarrangement 301 is operated to collectively illuminate two objects 350by the beam 301′ as an instruction to the user to collect these objects350. The resulting collective illumination of the objects 350 isintuitive to the user.

In some embodiments, the device 320 is configured to cause theillumination arrangement 301, based on the manipulation control rule R4,to provide a user instruction to actively guide a user on how to performa dedicated manipulation operation. The user instruction thus guides theuser on how to maneuver the object. The user instruction may be providedby an instructive image projected onto the object and/or by illuminatingthe object by a specific color representing the manipulation operation.These embodiments will facilitate for the user and prevent unintentionalerrors. In the example of FIG. 20D, the illumination arrangement 301 isoperated to illuminate an object 350 with a beam 301′ that defines animage 301″ that indicates how the object, for example a bolt, is to beturned by the user 1. In some embodiments, the device 320 monitors themanipulation operation and provides, by the selective illumination, auser instruction to stop the manipulation operation. In the example ofFIG. 20D, the device 320 may calculate the number of turns imposed onthe object 350 and provide the beam 310′ with a visual cue to instructthe user to stop turning the object, for example by changing theimage/color.

In some embodiments, the device 320 is configured to cause theillumination arrangement 301, based on the retrieval rule R5, to providea user instruction to actively guide a user to retrieve an object. Theobject may be detected, based on R5, to be at risk of leaving the fieldof vision of the user and/or be beyond reach of the user. This object isdenoted “potentially lost object”, PLO, in the following. The userinstructions may be provided by illuminating the object, optionally witha dedicated visual cue. These embodiments will reduce the risk that auser loses tools, materials or other equipment that are manipulated bythe user during EVA. An example is shown in FIG. 20E, where RLdesignates the reach limit of user 1 and IFV′ designates the field ofvision of user 1, and the user instruction is provided by illuminating aPLO 350 by the beam 301′. In FIG. 20E, IFV′ is bounded by a left-handlimit, IFVL, and a right-hand liming, IFVR. The PLO may be detected byevaluating the position of one or more objects in relation to RL and/orIFV′. Depending on implementation, the IFV′ may be independent of headpose and defined only in relation to the helmet 10 (cf. static field ofvision in Chapter 1), or be defined in relation to the head pose insidethe helmet 10 (cf. dynamic field of vision in Chapter 1). The reachlimit RL may be estimated based on the body pose of the user 1, byaccounting for the location of the user’s hands, the posture, theconstraints of the space suit 2, and the impact of low gravity. Thus, insome embodiments, the device 320 is configured to, based on R5, estimateRL and/or IFV′. RL may be estimated based on BPD, and dynamic field ofvision may be estimated based on HPD (cf. FIGS. 16 and 18 ). Staticfield of vision may be estimated based on an object property indicativeof helmet orientation, if determined in step 312 (FIG. 17 ). In afurther variant, IFV′ is determined based on the gaze direction of theuser, provided that the device 320 obtains gaze direction data, GDD,from a gaze tracking device (cf. 202 in FIG. 10 ). For example, IFV′ maybe set to correspond to the span of the central vision or the peripheralvison in relation to the gaze direction of the user, while alsoaccounting for blockage by the visor (cf. 12 in FIG. 15 ).

In some embodiments, the device 320 is configured to determine atrajectory of a respective object within the detection space. Thetrajectory may be determined as described in Chapter 1 with reference toFIG. 7B. Subject to R5, the device 320 may detect the PLO by comparingthe trajectory of the respective object to IFV′ and RL, respectively.Thus, even if an object is well within IFV′ and/or RL, it maynevertheless be identified as a PLO. In the example of FIG. 20E, atrajectory T1 is determined for object 350. Since the object 350 isclose to RL and IFVR and has a trajectory T1 crossing RL and IFVR, theobject 350 may be identified as a PLO.

In some embodiments, the device 320 is configured to cause theillumination arrangement 301, based on the work sharing rule R6, toprovide a user instruction to actively instruct a user to perform anoperation. The user may be selected among a plurality of users based onone or more properties of the users and/or based on one or moreproperties of the object(s) to be manipulated in the operation. The oneor more properties may be determined in step 312 (FIG. 17 ). It is to benoted that the monitoring sensor(s) 31 may be configured to also detectobjects that represent users and that such “user objects” may beassigned one or more object properties. These embodiments willfacilitate cooperation among users. In one example, the user may beselected based on the distance and/or movement direction between theobject(s) to be manipulated and the plurality of users. In anotherexample, the user may be selected in view of on-going or upcomingoperations performed by the users. In some embodiments, the userinstruction may be provided by illuminating the selected user. In otherembodiments, the user instruction is provided by illuminating theobject(s) to be manipulated by a visual cue that is associated with theselected user, for example a dedicated color, an image designating theuser, etc. In the example of FIG. 20F, the illumination arrangement 301of user 1 is operated to illuminate an object 350 by a beam 301′ with avisual cue designating user 1′ to manipulate the illuminated object 350.

In some embodiments, the device 320 is configured to cause theillumination arrangement 301, based on the anchor detection rule R7, toprovide a user instruction to actively guide a user to connect ananchoring device to at least one object, which is detected to be a safeanchoring point. The embodiments will improve the safety of the user. Inthe example of FIG. 20G, the object 351 is a safe anchoring point and isilluminated by beam 301′ to instruct the user 1 to connect an anchoringline 352 to the object 351.

In all of the foregoing examples, the visible light 301′ may begenerated with visual cues to enhance the user’s appreciation of therespective user instruction. A visual cue is a visual property and maycomprise, in any combination, an image or pattern of projected light, acolor of projected light, or a radiant intensity of projected light.Different visual cues may be applied to represent different userinstructions and/or different objects and/or different objectproperties. For example, a user assignment (FIG. 20F) may be indicatedby different visual cues for different users. Different temperatures ofobjects may be indicated by different visual cues. Objects at differentdistances to a user may be illuminated by different visual cues. ThePLOs may be indicated by a dedicated visual cue. Associated objects(FIG. 20C) may be illuminated with the same visual cue. Objects may beilluminated by different visual cues depending on if they are deemed topose a danger to the user or not, and/or to indicate different degreesof danger (“risk levels”). Objects may be illuminated by differentvisual cues depending on the direction of relative movement between auser and the object.

In some embodiments, the selection of objects to be illuminated maydepend on the speed of the user or the relative speed between the userand the respective object. In an example, one or more rules in the ruledefinition [R] may be configured to cause the device 320 to illuminatefiner-detailed objects if the (relative) speed is low, for example belowa first speed limit, and to illuminate only the largest objects ofrelevance if the (relative) speed is high, for example above a secondspeed limit.

In some embodiments, a user is given the option to reject a userinstruction provided by the device 320. For example, the user may rejecta user instruction by operating a manual control (cf. 112 in FIG. 2 ) orby uttering a dedicated voice command. This may cause the device 320 toperform steps 311-314 anew to generate a new user instruction. In theexample of FIG. 20A, the user 1 may reject an illuminated safe area 350.In the example of FIG. 20F, the user 1′ may reject to be assigned anoperation. In the example of FIG. 20G, the user 1 may reject anilluminated safe anchoring point 351.

In some embodiments, the device 320 is configured to instruct the userto pay attention to objects outside the field of vision by projecting auser instruction on a surface within the user’s field of vision. Theuser instruction may be provided by any type of visual cue that causesthe user to look around for an object of relevance. The visual cue mayindicate a direction for the user to look/turn, for example in cleartext or by an arrow. The field of vision may be determined as describedwith reference to FIG. 20E. The objects outside to the field of visionmay result in a user instruction if they are deemed to pose a danger tothe user, for example based on movement direction, trajectory, shape,color, composition, etc.

One advantage of some embodiments described in the foregoing that theilluminated objects, and the associated user instructions, areinherently visible to all users that are present at the detection space.Thus, users may cooperate even if a conventional communication system isunavailable.

Another advantage of some embodiments, compared to AR based solutions,is that user instructions are provided without bloating or clutteringthe field of vision of the user.

Further, some embodiments enable the user to be instructed with respectto object properties that are imperceptible to the human eye, forexample objects that are too hot or too cold to touch, or slippery orunstable ground.

Further, some embodiments enable tracking of objects in low gravity andprovide early warning when an objects is about to move too far away fromthe user.

In the following, clauses are recited to summarize some aspects andembodiments as disclosed in the foregoing.

C1. A device for providing assistance during extravehicular activity,said device comprising: an input (320A) for first data (ESD) indicativeof one or more objects in a detection space; and processor circuitry(101) configured to: obtain the first data (ESD) on the first input(320A); process the first data (ESD) for determination of one or moreobject properties for the one or more objects; evaluate, based on theone or more object properties, the one or more objects in relation to aset of rules ([R]) to determine one or more user instructions (UI); andcause an illumination arrangement (301) to provide the one or more userinstructions (UI) by selective projection of light (301′) in relation tothe one or more objects.

C2. The device of C1, wherein the one or more object propertiescomprises a classification of a respective object into one or moreobject categories.

C3. The device of C1 or C2, wherein the processor circuitry (101) isconfigured to cause the illumination arrangement (301) to provide theone or more user instructions (UI) by selective projection of the light(301′) onto at least one of the objects.

C4. The device of any preceding clause, wherein the processor circuitry(101) is configured to cause the illumination arrangement (301) torepresent different user instructions by different visual properties ofthe light (301′).

C5. The device of any preceding clause, wherein the set of rules ([R])comprises at least one risk detection rule (R1) for identification of atleast one object (340, 341) that poses an actual or potential danger,and wherein the processor circuitry (101) is configured to, based on theat least one risk detection rule (R1), cause the illuminationarrangement (301) to provide a user instruction to avoid the at leastone object (340, 341) by the selective projection of the light (301′).

C6. The device of C5, wherein the processor circuitry (101) isconfigured to repeatedly process the first data (ESD), evaluate the oneof more objects in relation to the at least one risk detection rule (R1)and cause the illumination arrangement (301) to provide the userinstruction to avoid the at least one object (340, 341), to therebyilluminate a safe path (360) in relation to the at least one object(340, 341).

C7. The device of C5 or C6, wherein the at least one risk detection rule(R1) is configured to identify the at least one object (340, 341) basedon its temperature.

C8. The device of any preceding clause, wherein the set of rules ([R])comprises at least one task control rule (R2) for identification of anordering among a plurality of objects for use in performing a task, andwherein the processor circuitry (101) is configured to, based on the atleast one task control rule (R2), cause the illumination arrangement(301) to provide a user instruction, by the selective projection of thelight (301′), to use the objects in accordance with the ordering.

C9. The device of C8, wherein the processor circuitry (101) isconfigured to cause the illumination arrangement (301) to sequentiallyproject the light (301′) onto the objects in accordance with theordering.

C10. The device of any preceding clause, wherein the set of rules ([R])comprises at least one completion rule (R3) for identification when atask is completed, and wherein the processor circuitry (101) isconfigured to, based on the at least one completion rule (R3), cause theillumination arrangement (301) to collectively project the light (301′)onto the plurality of objects as a user instruction to collect theplurality of objects.

C11. The device of any preceding clause, wherein the processor circuitry(101) is configured to estimate at least one of a field of vision (IFV′)of an individual or a reach limit (RL) of the individual, wherein theset of rules ([R]) comprises at least one retrieval rule (R5) foridentification of at least one object (350) which is at risk of leavingthe field of vision (IFV′) and/or be beyond the reach limit (RL), andwherein the processor circuitry (101) is configured to, based on the atleast one retrieval rule (R5), cause the illumination arrangement (301)to provide a user instruction to retrieve the at least one object (350)by the selective projection of the light (301′).

C12. The device of C11, wherein the processor circuitry (101) isconfigured to estimate a trajectory (T1) of the one or more objects, andwherein the at least one retrieval rule (R5) is configured to identifysaid at least one object by comparing the trajectory (T1) to the fieldof vision (IFV′) and/or the reach limit (RL).

C13. The device of C11 or C12, which comprises a further input (320B)for measurement data (BPD) indicative of a body pose of the individual,wherein the processor circuitry (101) is configured to estimate thereach limit (RL) of the individual based on the body pose.

C14. The device of any one of C11-C13, which comprises a further input(320C) for measurement data (HPD, GDD) indicative of a head pose and/ora gaze direction of the individual, wherein the processor circuitry(101) is configured to estimate the field of vision (IFV′) of theindividual based on the head pose and/or the gaze direction.

C15. The device of any preceding clause, wherein the set of rules ([R])comprises at least one manipulation control rule (R4) for identificationof a manipulation operation to be performed on an object, and whereinthe processor circuitry (101) is configured to, based on the at leastone manipulation control rule (R4), cause the illumination arrangement(301) to provide a user instruction on how to perform the manipulationoperation by the selective projection of the light (301′).

C16. The device of any preceding clause, wherein the set of decisionrules ([R]) comprises at least one work sharing rule (R6) for selectionof an object, which corresponds to an individual (1; 1′), to perform anoperation, and wherein the processor circuitry (101) is configured to,based on the at least one work sharing rule (R6), cause the illuminationarrangement (301) to provide a user instruction to the individual (1;1′) to perform the operation by the selective projection of the light(301′).

C17. The device of any preceding clause, wherein the set of decisionrules ([R]) comprises at least one anchor detection rule (R7) foridentification of at least one object (351) that is a safe anchoringpoint, and wherein the processor circuitry (101) is configured to, basedon the at least one anchor detection rule (R7), cause the illuminationarrangement (301) to provide a user instruction to connect an anchoringdevice (352) to the at least one object (351) by the selectiveprojection of the light (301′).

C18. The device of any preceding clause, further comprising at least oneof a sensor arrangement (108) configured to generate the first data(ESD), or the illumination arrangement (301).

C19. A space suit for extravehicular activity, said space suit (2)comprising a device according to any preceding clause.

C20. A computer-implemented method of providing assistance duringextravehicular activity, said computer-implemented method comprising:

-   obtaining (311) first data indicative of one or more objects in a    detection space;-   processing (312) the first data for determination of one or more    object properties for the one or more objects;-   evaluating (313), based on the one or more object properties, the    one or more objects in relation to a set of rules to determine one    or more user instructions; and-   causing (314) an illumination arrangement to provide the one or more    user instructions by selective projection of light in relation to    the one or more objects.

C21. A computer-readable medium comprising instructions which, wheninstalled on a processor (401), causes the processor (401) to performthe method of C20.

The structures and methods disclosed herein may be implemented byhardware or a combination of software and hardware. In some embodiments,such hardware comprises one or more software-controlled computersystems. FIG. 21 schematically depicts such a computer system 400, whichcomprises one or more processors 401, computer memory 402, and aninterface device 403 for input and/or output of data. Depending onimplementation, the computer system 400 may be included in an SSS, inany external computing resource in communication with an SSS, or anyother computing resource. The interface device 403 may be configured forwired and/or wireless communication. The processor(s) 401 may, forexample, include one or more of a CPU (“Central Processing Unit”), a DSP(“Digital Signal Processor”), a microprocessor, a microcontroller, anASIC (“Application-Specific Integrated Circuit”), a combination ofdiscrete analog and/or digital components, or some other programmablelogical device, such as an FPGA (“Field Programmable Gate Array”). Acontrol program 402A comprising computer instructions is stored in thememory 402 and executed by the processor(s) 401 to perform any of themethods, operations, functions or steps exemplified in the foregoing. Asindicated in FIG. 21 , the memory 402 may also store control data 402Bfor use by the processor(s) 401. The control program 402A may besupplied to the computer system 400 on a computer-readable medium 410,which may be a tangible (non-transitory) product (e.g. magnetic medium,optical disk, read-only memory, flash memory, etc.) or a propagatingsignal.

While the subject of the present disclosure has been described inconnection with what is presently considered to be the most practicalembodiments, it is to be understood that the subject of the presentdisclosure is not to be limited to the disclosed embodiments, but on thecontrary, is intended to cover various modifications and equivalentarrangements included within the spirit and the scope of the appendedclaims.

Further, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, parallel processing may beadvantageous.

As noted, any and all combinations of the above-described concepts andembodiments are possible and may provide synergies. A few non-limitingexamples are presented below. The embodiments in Chapter 1 may becombined with the embodiments in Chapter 3 to improve the ability ofusers to perceive dangers in their surroundings, by combiningaudio-based and illumination-based assistance. The illuminationarrangement, as included in the embodiments in Chapter 3, may be used asa feedback device for the embodiments in Chapter 2, to provideperformance-related feedback to a user. The next action as determined bysome embodiments in Chapter 2 (cf. FIG. 13D) may be presented as anillumination-based user instruction by the embodiments in Chapter 3.Alternatively or additionally, the next action may be presented as aspatialized audio signal by use of the embodiments in Chapter 1. Theaudio signal may, by its spatial origin, inform the user on how and/orwhere to move in the next action.

What is claimed is:
 1. A device for providing assistance duringextravehicular activity, said device comprising: an input for first dataindicative of one or more objects in a detection space; and processorcircuitry configured to: obtain the first data on the first input;process the first data for determination of one or more objectproperties for the one or more objects; evaluate, based on the one ormore object properties, the one or more objects in relation to a set ofrules to determine one or more user instructions; and cause anillumination arrangement to provide the one or more user instructions byselective projection of light in relation to the one or more objects. 2.The device of claim 1, wherein the one or more object propertiescomprises a classification of a respective object into one or moreobject categories.
 3. The device of claim 1, wherein the processorcircuitry is configured to cause the illumination arrangement to providethe one or more user instructions by selective projection of the lightonto at least one of the objects.
 4. The device of claim 1, wherein theprocessor circuitry is configured to cause the illumination arrangementto represent different user instructions by different visual propertiesof the light.
 5. The device of claim 1, wherein the set of rulescomprises at least one risk detection rule for identification of atleast one object that poses an actual or potential danger, and whereinthe processor circuitry is configured to, based on the at least one riskdetection rule, cause the illumination arrangement to provide a userinstruction to avoid the at least one object by the selective projectionof the light.
 6. The device of claim 1, wherein the processor circuitryis configured to repeatedly process the first data, evaluate the one ofmore objects in relation to the at least one risk detection rule andcause the illumination arrangement to provide the user instruction toavoid the at least one object, to thereby illuminate a safe path inrelation to the at least one object.
 7. The device of claim 1, whereinthe at least one risk detection rule is configured to identify the atleast one object based on its temperature.
 8. The device of claim 1,wherein the set of rules comprises at least one task control rule foridentification of an ordering among a plurality of objects for use inperforming a task, and wherein the processor circuitry is configured to,based on the at least one task control rule, cause the illuminationarrangement to provide a user instruction, by the selective projectionof the light, to use the objects in accordance with the ordering.
 9. Thedevice of claim 8, wherein the processor circuitry is configured tocause the illumination arrangement to sequentially project the lightonto the objects in accordance with the ordering.
 10. The device ofclaim 1, wherein the set of rules comprises at least one completion rulefor identification when a task is completed, and wherein the processorcircuitry is configured to, based on the at least one completion rule,cause the illumination arrangement to collectively project the lightonto the plurality of objects as a user instruction to collect theplurality of objects.
 11. The device of claim 1, wherein the processorcircuitry is configured to estimate at least one of a field of vision ofan individual or a reach limit of the individual, wherein the set ofrules comprises at least one retrieval rule for identification of atleast one object which is at risk of leaving the field of vision and/orbe beyond the reach limit, and wherein the processor circuitry isconfigured to, based on the at least one retrieval rule, cause theillumination arrangement to provide a user instruction to retrieve theat least one object by the selective projection of the light.
 12. Thedevice of claim 11, wherein the processor circuitry is configured toestimate a trajectory of the one or more objects, and wherein the atleast one retrieval rule is configured to identify said at least oneobject by comparing the trajectory to the field of vision and/or thereach limit.
 13. The device of claim 11, which comprises a further inputfor measurement data indicative of a body pose of the individual,wherein the processor circuitry is configured to estimate the reachlimit of the individual based on the body pose.
 14. The device of claim11, which comprises a further input for measurement data indicative of ahead pose and/or a gaze direction of the individual, wherein theprocessor circuitry is configured to estimate the field of vision of theindividual based on the head pose and/or the gaze direction.
 15. Thedevice of claim 1, wherein the set of rules comprises at least onemanipulation control rule for identification of a manipulation operationto be performed on an object, and wherein the processor circuitry isconfigured to, based on the at least one manipulation control rule,cause the illumination arrangement to provide a user instruction on howto perform the manipulation operation by the selective projection of thelight.
 16. The device of claim 1, wherein the set of decision rulescomprises at least one work sharing rule for selection of an object,which corresponds to an individual, to perform an operation, and whereinthe processor circuitry is configured to, based on the at least one worksharing rule, cause the illumination arrangement to provide a userinstruction to the individual to perform the operation by the selectiveprojection of the light.
 17. The device of claim 1, wherein the set ofdecision rules comprises at least one anchor detection rule foridentification of at least one object that is a safe anchoring point,and wherein the processor circuitry is configured to, based on the atleast one anchor detection rule, cause the illumination arrangement toprovide a user instruction to connect an anchoring device to the atleast one object by the selective projection of the light.
 18. Thedevice of claim 1, further comprising at least one of a sensorarrangement configured to generate the first data, or the illuminationarrangement.
 19. A space suit for extravehicular activity, said spacesuit comprising a device according to claim
 1. 20. Acomputer-implemented method of providing assistance duringextravehicular activity, said computer-implemented method comprising:obtaining first data indicative of one or more objects in a detectionspace; processing the first data for determination of one or more objectproperties for the one or more objects; evaluating, based on the one ormore object properties, the one or more objects in relation to a set ofrules to determine one or more user instructions; and causing anillumination arrangement to provide the one or more user instructions byselective projection of light in relation to the one or more objects.